From ad57c9ee2585c9d03dc6f7262af9efafcad9bbeb Mon Sep 17 00:00:00 2001 From: Yevhenii Date: Wed, 21 Oct 2026 20:46:27 +0300 Subject: [PATCH] fix(bridge): fail a missing image snapshot in CI mode instead of writing it In Jest's CI mode, toMatchImageSnapshot fails when its snapshot does not exist and saves the received image beside it as -actual.png, where it wrote the snapshot and passed. The Jest session hands the bridge Jest's updateSnapshot state; a bridge context that names none writes a missing snapshot as before. The received and diff images take the whole snapshot name, and toMatchImageSnapshot refuses .not, which would turn the failure into a pass. The playground commits the baselines CI's Android emulator, iOS simulator and Chromium capture for its screenshot and out-of-bounds specs and ignores the failure images saved beside them. The UI testing guide and the CLI's UI skill say how snapshots are created in CI mode, and the CLI reference lists --ci. --- .gitignore | 3 + .../version-plan-1792417138768.md | 5 + .../android/orange-square-element-only.png | Bin 0 -> 865 bytes .../android/out-of-bounds.png | Bin 0 -> 3500 bytes .../chromium/orange-square-element-only.png | Bin 0 -> 323 bytes .../chromium/out-of-bounds.png | Bin 0 -> 663 bytes .../ios/orange-square-element-only.png | Bin 0 -> 4444 bytes .../__image_snapshots__/ios/out-of-bounds.png | Bin 0 -> 30118 bytes .../src/__tests__/image-snapshot.test.ts | 184 ++++++++++++++++++ packages/bridge/src/image-snapshot.ts | 28 ++- packages/bridge/src/server.ts | 8 +- packages/bridge/src/shared.ts | 4 + packages/cli/skills/ui.md | 3 +- packages/jest/src/__tests__/bridge.test.ts | 66 ++++++- packages/jest/src/harness-session.ts | 5 +- .../__tests__/to-match-image-snapshot.test.ts | 62 ++++++ .../expect/matchers/toMatchImageSnapshot.ts | 4 + packages/runtime/src/globals.ts | 3 +- website/src/docs/api/cli.mdx | 3 +- website/src/docs/guides/ui-testing.mdx | 8 + 20 files changed, 372 insertions(+), 14 deletions(-) create mode 100644 .nx/version-plans/version-plan-1792417138768.md create mode 100644 apps/playground/src/__tests__/normal/ui/__image_snapshots__/android/orange-square-element-only.png create mode 100644 apps/playground/src/__tests__/normal/ui/__image_snapshots__/android/out-of-bounds.png create mode 100644 apps/playground/src/__tests__/normal/ui/__image_snapshots__/chromium/orange-square-element-only.png create mode 100644 apps/playground/src/__tests__/normal/ui/__image_snapshots__/chromium/out-of-bounds.png create mode 100644 apps/playground/src/__tests__/normal/ui/__image_snapshots__/ios/orange-square-element-only.png create mode 100644 apps/playground/src/__tests__/normal/ui/__image_snapshots__/ios/out-of-bounds.png create mode 100644 packages/bridge/src/__tests__/image-snapshot.test.ts create mode 100644 packages/runtime/src/__tests__/to-match-image-snapshot.test.ts diff --git a/.gitignore b/.gitignore index b21ea997..780fb710 100644 --- a/.gitignore +++ b/.gitignore @@ -112,6 +112,9 @@ yarn-error.log # testing /coverage .harness/ +# Images a failed match saves beside the playground's committed image snapshots +/apps/playground/**/__image_snapshots__/**/*-actual.png +/apps/playground/**/__image_snapshots__/**/*-diff.png # Prebuilt XCTest agent artifacts, packed by the platform-apple `prepack` script /packages/platform-ios/xctest-agent-prebuilds/ vite.config.*.timestamp* diff --git a/.nx/version-plans/version-plan-1792417138768.md b/.nx/version-plans/version-plan-1792417138768.md new file mode 100644 index 00000000..4e38be20 --- /dev/null +++ b/.nx/version-plans/version-plan-1792417138768.md @@ -0,0 +1,5 @@ +--- +__default__: minor +--- + +In Jest's CI mode (on by default in CI), `toMatchImageSnapshot` fails when its snapshot does not exist instead of writing it and passing, and saves the received image beside it as `-actual.png`. A project whose CI relied on missing snapshots being written commits those images as its snapshots, or runs with `--ci=false` on a machine with the comparing device. Like Jest's own snapshot matchers, `toMatchImageSnapshot` cannot be used with `.not`. diff --git a/apps/playground/src/__tests__/normal/ui/__image_snapshots__/android/orange-square-element-only.png b/apps/playground/src/__tests__/normal/ui/__image_snapshots__/android/orange-square-element-only.png new file mode 100644 index 0000000000000000000000000000000000000000..98f46435d3de6781f1ae02648ab02924e292de63 GIT binary patch literal 865 zcmeAS@N?(olHy`uVBq!ia0y~yU}OVf4j{?UZfb1VJUX< z4B-HR8jh3>pr$ZS7srr_Id5+n2C_JcxL%aYdBBjA6tiaTOw-M26B=gR6XEdOm4DTe zEAGeYf40vgiW&Zx9JhG4+MVI{&N(~>@{8qGt20RNlE40n^FZ3U%HEF=dzo&Wdwu;C z>w!Ew`(uy)=6>Z$FuuS2Hdy|%RQq?9564fxVgxz`3jQlLvB202lUydi*$$I9gy0+x zAx9=S$5Y7z&Q|bb;e>NkIu+n1H>e1zz$GTAj9LN?jM3nNriszi1r3PNd;%*aMvE0_ zNQ{=k(0~}NPoM!YTKz%;VzeOv4T#Z(1T-K<8xqif7;Q*E1A@WRUt@s~_l-ZFuYp87 MUHx3vIVCg!025F&%K!iX literal 0 HcmV?d00001 diff --git a/apps/playground/src/__tests__/normal/ui/__image_snapshots__/chromium/orange-square-element-only.png b/apps/playground/src/__tests__/normal/ui/__image_snapshots__/chromium/orange-square-element-only.png new file mode 100644 index 0000000000000000000000000000000000000000..64c61141ca118bff5c3297938204bb56da77ac8a GIT binary patch literal 323 zcmeAS@N?(olHy`uVBq!ia0vp^DImxdHmX`P}SB*2{BuV)GuyqB6C5vD+6xWV+=zVk0;>Szm) r1b;9BL*V~|r&{}fVZa6o17?O97g+CXgTe~DWM4fGrd}x literal 0 HcmV?d00001 diff --git a/apps/playground/src/__tests__/normal/ui/__image_snapshots__/chromium/out-of-bounds.png b/apps/playground/src/__tests__/normal/ui/__image_snapshots__/chromium/out-of-bounds.png new file mode 100644 index 0000000000000000000000000000000000000000..f2af63da6ec4d0106fb43101afdb26dfbb23a59f GIT binary patch literal 663 zcmeAS@N?(olHy`uVBq!ia0y~yV15B)r*JR<$yw38y$lRYKAtX)Ar*7pTyxBI4isQH zxHU!W;nVDQiW9{RIA=N-%*f}eF7XbPNL#Wm{_wT8JkiHDZCm%+?9CsGhqu1Jmz-wz zuYKFhwtP$ zB8Y9qZs`@TJ3Qj9EZ|)Jc4CIP`Jw}y%p;U?=GCd(E%>juIy0-zUzFZ;KVnH zA_T>!omo>C63m;wj3mmP-BCY7^`C5$4tkkvJGu-4OLc}(73EN|WVZskTa6bHQ z`t>V=)f`BE6`3}oA3ka^P)L=}v|%+{S%1XXwvtKHQ7nU%G^rsBPjN6qji_|GW}fIaP)e)26n^7$0>b8KsrW+b0Q`DCXAbQmtfX zsbt|vX^1!oWn9$9PZN*2i3`*k{ilwCWCrcF(ku?eN-?&dDZyssG%|c_q^1RtawRAe zLyKE6(gw2R>;iD6GH-HneCYN(g=i>+hxc!)A>$7zxMv6pPZ91&W#+O%^G~7u`lpGu z=TdzlYbtT1TtrY>x{S6siz`^mb=pkXjE9#QrgVpSbnERX(EM`7+iloRapI(h_7Zkh z7-s=X5Ww>2gaxY9rMiREhYcEJ!3rhYA1sm=iKI`-5*{8IpH{EeO75KQAR{Q@$%Z#3 z44a79kDQ8a9KMQF=L@dBV1v`x7zro%cESxVUE3fDx0|H!&5F{QdG&e(T}amADdt5u z67UW!o`W)iM~zMncjb72uYYh*C8mH0`9|^9+}YU@aNQ>16`?FVP_{2rb;~IO3R^RbSr@(a| zSf>~%QT_%*ttNeEx=4GSqmeXSBk7>;0|fxg^3qZlaLpT73{<-YsI7DT9MM+~)bnWH zt-HXFb)>4@ls<*H{j4WHefXf#9BA971*v06L#syHs7BkEag8rn#DWF#n{kQvVVr=r zCMz}Uo}q2ZARwrLqs5do&|n80Ad;)}@RrW_$Ew{wkI}%QZfh-P=3eSvpwSmbmIw!3 z+QZyTWqCmM7wxztZ;rPD@(FtH4vou4Qs2gd$Tk(k+z_L`siXGxA zEg-D5+OXoAK5AgS(J1VhnYssNi=mdN=z>ZYm^g^oO@lqpRbK}G%9d*}bwyQWYna9* zeKEd$83r~R92g8>C|+d3eEp!ON@HK38u{>`f3E@zywJ|5M`Je^Lw+Iwhn|wE3?Rbq zpv_~|v-A>>VIXyv>>2KcU{ba8gp$-+1f}Y{-8?qXm;ehJh|$8oE`iR_FU=1EoKY${tJO%(D48O literal 0 HcmV?d00001 diff --git a/apps/playground/src/__tests__/normal/ui/__image_snapshots__/ios/out-of-bounds.png b/apps/playground/src/__tests__/normal/ui/__image_snapshots__/ios/out-of-bounds.png new file mode 100644 index 0000000000000000000000000000000000000000..33126c3118365b340e1cb5f9468d825ff55554e7 GIT binary patch literal 30118 zcmeHQYgiLk8crA-0}Ki%NWj)eb=%@C6f2ZgjV#nlTZ&duplYBBR58X?iinq_qM~do zy=b+9BKz#>3JB@~QA7i3wXK3~7mX!g(6(3+utmAl0()kXnMvEz#3woPZ}h<+bT} z8A}(;OnBoBfj{_OBp4v{5Mbm}z+ZvTM=*f;ULcqy^ku%EBYe*E90!45+q(is({p0L zBl$-MzQC(Zk650NeaAcp`q?g;JdFNcIF@`_wA1`L@ZhAJ5xrU<7#U1{3Bz{>$iWYX zzZ>@ET<{1!kovP5{QDgEA|Jt5NZPjx3s->0@R?!L=B^dq*H;yN|Ekx3s+DiX&P@Dv zaO={cDWa2k_@|m1kPi|=LN$U;7aPJx3B}GHaJePm> zsamY-7e6F_Er~Pqq%G>dtPUZ+tMR%Cj^2I z%5y7?`A7HXLVOB#yEC`gIj;D{brD47I&5}vzo2SUMwRv~?mWBT#T~_Bxgk)Z7}g!A zm0gheEnB!IE;m6dz?{bp_}uqb@g;9JP5Z9iZg8Qah_LFYSb2iW(5%OA^afzJ+nPLr z_6ad3tU9hqiV<^(%!&bRgfu%q^1!PHyxn<~azXHA0p^MADya(z*AY^QPv6JUVEpO~ z_0*_Y&;4uYTZHt%2;UiDMLWB7tK(JbcDG~~DVV(^Z=L^;xPlaL&C*Dx*rK;Gt_ZNb zC3UTJ3TdKBJwta7UtYcke8%YH(g^oS7?HU({G4f4x{lopxdNtR=z_2Iy_$CCrL>4! zRp0+a&40YiF#c!o`b^=qlB7LC+#e4+mmi0F9|ogbxzaNNjK&pnNG(sLu0CnO#NS>0 zqST*LlE`)F--N{;R9>gnRHpNFN*~d(WS6jhR($L#qH`1G-F<3NkOsUP5k}0dj%pbi z7pPTfVhuF{OpJX5hS8vCPf)36>6YdCCF9<1u31~f^1OJNZkz&K-QR`twjJNbUfm_{ zlG3?lUM))=offJeZ*=@fb0cJOK?`&3mokbJI)%QZ6%2LLe^uIs)&@fA79jEQa-|lP zo}(qz*Bizr6(wl3tFY#*LGugSd`}+#lJLP_h--N!w6oAVcxB#CF{Qm}VA3URUghA$ zUoaCa(x+bP`()bD{uLcd4QB)0oMIaqTRRnC?lOw5KVO}oVy4vb+{Mjt{Y}~sC;WxL z$kM|xz%Jqq>!o9IHi_M862WHmkIhOszOGUG$oF7xZ@QD{^c6|GY~olSFeN*LtGa52 zIV=g3=f%kkZR7`JGV_z?Q5JJZv`+{wxImP4>-ZL3f>N*T$$w{pFc9n-av%k63p2qi ze{kbSYc1GN!nl>A?t30=zj!Y?_rNrIA~Gw=S7zU`zAAD#KCE=StR=w4cT08AgBQHRECn$10e3h1YbRlN zBeP>t+wc1+z)yT|zqo?>iiSpDAFP!b7hWarUMn5@`Qm4*WC{2YuX!${je<=uJ>!0K zfU;3l^v~m^3!W`#b`>4!dd6)KWy<++*G#6IdWN!Y3-*xh6dQi&@$PZwCvCoi*9?0q z-D!_nEWIjKPTAvGbCwV_2jV`(^Xd;YpBG@;56@FgIqZ-eDA&5SO}+iYEPZlr;f{*b z_uNScakXR=()F)+dtE91L@o)$=4Bn(LK+6M&}F;7x*nffxIaje)3{R>oJ_(LjRlgn z#vbrFI5CkB>8TlV$LR6VRw?^SvggS5*fO_z;zQry#cS@5mg5Op**TSVgYRzTx8#B( zosL?uY5x#_z4=kXU-Fzw0JsEcDynk9KE%OFS7h3>m2NB4FJr_K)*MvIbyFRhO2pD^ zY2?Y1H$%GIU&~y9Uv$l^uze}O5{+;_o_Il<#+ZaU?0^2U5p#uQVyS71_in>R12oTF zc>W4GblIrnl=O0P*E06Tpd>M3@Ye8-mQmi9F3aljm&E~tRd-YsD*Kkof)519m;XC? z-1RT-rD4@S_KGsjfp@ExJ+CN@3YdhI+dGaecSF~JgMpjHfZ27%H^8p9QFa{^-+m`s zdy@dCx?5`d`q4(qb8MqfoIOnZ!HKn_1Q_p-vK`r7<6^1h%b5_;=^@1^K+}&?lK8Iv zVtHL>^Gg9*aNc_64L%~M2k#tKdN{; zeAm9UFr0CHEqDbvq=}0P%BI$C7vdhX3j`Y?2j~*Cs`{@RjHk8Db|G92n?~JK~c1I=fHFMJt-c~Tj9ApGv7&Q}M{Q-N)!snoj~$+10Fu~T&OM7-g&F!r^j%a27w)%v6Wn1kN~AB3b0Rs}Rv6YILMfW5u~d*fGSn?qg=ZH}tdU&=)6_i1I872|D!F09 z7>2@F;NFsX8K+m?3{4IURv6Y_JNB7MyoaPkETF^ILNOtAGsPjmrnQks)eJmQ-Se5S zY5f;aI;bdayf{(~);Jq@1h%s&l4a$)Tmy`P_Ad~l6rgMS3XF7(^_38#)cFR7RNVX0 zpHPtYsJpZD%X9bFrPx8GUf2DSnLT+7UgHZJg<+kvrG3~3fmZ|USqn2jhmooB&+KNl zU^8cq4@*>WiBiB3o;0@UKP33TECU5f$xLhwI*g2S)kfXG4mt>6aEjlOTEL+H#80a5 zJmt|9b0{&T!FH^LIo@WT1{yNO06J_f5TKM}Tm9QxzM=dwnA3^~r5J^VRu9Z*2!&UE z>q)WAcCcl27TA#?Y${Pwnv^~2LGXyn&drPQj1Do(;X_2GfLCes#<)W%EB-eEl(OW1 z8#?c8814~FDI)ebORU%;gHS?JPdFtyKxwobdEZD}z9rRBBTu;jUk0w`pg<{^;mnnJ zMsJLf2a%%`4d_rxn10filHF`~F@ROjS_@2-(p0V+vCXlD)I{u}gw30yn|}RTrJMp) z2vN9v3|r$2rGVOg&cmp-L$DSRO3~gJl-Sld}*6k`OM zk&S9CtF}N{2r8EkxPoEB@`3Bh5lWG4ZN8DMapv;T!6zsMQQj)Ly2}(cv0budy)oD* zw2$Q1Z%yz?%zFo52v_bm0y1S$J4c05Ku}#^43u!@^3m}$7}$!oZK1+}_Bcua1~-IV z+V6=qyTTMgUw!JBHKW@~BWmxR;!IvcS}(phw2hfhYPlUJJ&aBE?jjW0fqR!`1>p zM8-^Qfw2%&WQrD4IYTKc;cpbe6df#qQV_$L*n?wM6{oC=xKYo9V!Scfk@5E6gx=o# zRd@)Qt7vnkvt{>NX@5kD1<3GSOTRWRU67f=igq`^04rz zFK6-^nurVw=+KcMNy7W{)KpX`WjTEfi7-X8H5k}RhIL2*AvA=_a~akYsMRKN7XC&Nk+G7xhREfky)h^wM1xY6-ER~w zpQ(rlW);Hn`ADi6%mh^kB}5XDVc8mIWQqxvphl_4y;V~|O=*AskX^H$0R^R@WI<^w zUrx%GlQPAyd^ss!PRf^)(q*eaDB;UV`EpXriaTFU%9oQ)BpXri<)nN$>63PWFstK0 zm0Qf0lbRZ*9OlbO`EpXGF9$2XLe&#Qg<4Dm6_$d5q5{H7z8jLFcMEzPXfJ{!d4bK4 zfMrpl$=b7`$ss7zqUBep1cayp!b(0XUruVNj}!7jEz6pm{YCFq;%SIbiix0d6&AFd z@5YytK6yJQ1Sn-Go`xzcXgQx1Mb@5K^ABid58cquzN}onoYd43&{D@5v+O5W3I?j& zVm5-x6`5k#8Z;P%B)6C^Cw;Q!e^%mYh{zOEpa4t3K!H+L^4*XWz0(PfsKbJ0%_P|j z2^iL*$=Wjmbm&OzCsR!J6PGAuIiD4Y%g2|K(rw`AhH4~m!+K*ffDSt`5abr~<)kz( zusbw{qEL(R#<)W%EABU6PD=M5X4x8NWQs2*r6p~qoio+Z`j0&f>E>Nr<`%PpDpzER zVQWyu7D;X~UrzdDZEyK%Mp%$Jk?rhzY= zzzjPvP~;Z#<)o&YhtR#u0YirNd*{nZpY*O2wAuo5G5B&)^P6k%<)o4H{eWn_0@k)3 zXcTnwE?9*Vs)!6LsKSbakQ8cJ41e?Gr0k7z_;OOn9i?D5&#@BzM&j~Wx)TClPWps* zIzfd}d^zbKZxjc;y_aQq4N-3n8qi@U247A}->}6>_?s^$HM{MdW%nC}%g2|KKB3FU k1WTaNRy*1dljWpq53QfyJz+O_uT{a!@QARZq4M|t51eFSYXATM literal 0 HcmV?d00001 diff --git a/packages/bridge/src/__tests__/image-snapshot.test.ts b/packages/bridge/src/__tests__/image-snapshot.test.ts new file mode 100644 index 00000000..0b54e82e --- /dev/null +++ b/packages/bridge/src/__tests__/image-snapshot.test.ts @@ -0,0 +1,184 @@ +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { PNG } from 'pngjs'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { matchImageSnapshot } from '../image-snapshot.js'; + +const createPng = ({ width = 2, height = 2, fill = 255 } = {}): Buffer => { + const png = new PNG({ width, height }); + png.data.fill(fill); + return PNG.sync.write(png); +}; + +describe('matchImageSnapshot', () => { + let directory: string; + let testFilePath: string; + let receivedPath: string; + let snapshotPath: string; + + beforeEach(async () => { + directory = await mkdtemp( + path.join(os.tmpdir(), 'harness-image-snapshot-') + ); + testFilePath = path.join(directory, 'screen.harness.tsx'); + receivedPath = path.join(directory, 'received.png'); + snapshotPath = path.join( + directory, + '__image_snapshots__', + 'android', + 'square.png' + ); + await writeFile(receivedPath, createPng()); + }); + + afterEach(async () => { + await rm(directory, { force: true, recursive: true }); + }); + + it('writes a missing snapshot and passes when new snapshots may be written', async () => { + const result = await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'square' }, + 'android', + 'new' + ); + + expect(result.pass).toBe(true); + expect(await readFile(snapshotPath)).toEqual(await readFile(receivedPath)); + expect( + existsSync(path.join(path.dirname(snapshotPath), 'square-actual.png')) + ).toBe(false); + }); + + it('writes a missing snapshot and passes when snapshots are updated', async () => { + const result = await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'square' }, + 'android', + 'all' + ); + + expect(result.pass).toBe(true); + expect(await readFile(snapshotPath)).toEqual(await readFile(receivedPath)); + expect( + existsSync(path.join(path.dirname(snapshotPath), 'square-actual.png')) + ).toBe(false); + }); + + it('fails a missing snapshot without writing it in CI mode, and saves the received image beside it', async () => { + const actualPath = path.join( + path.dirname(snapshotPath), + 'square-actual.png' + ); + + const result = await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'square' }, + 'android', + 'none' + ); + + expect(result.pass).toBe(false); + expect(result.message).toContain( + `New snapshot was not written: ${snapshotPath} does not exist` + ); + expect(result.message).toContain(`Received image saved at ${actualPath}`); + expect(existsSync(snapshotPath)).toBe(false); + expect(await readFile(actualPath)).toEqual(await readFile(receivedPath)); + }); + + it('compares an existing snapshot in CI mode', async () => { + await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'square' }, + 'android', + 'new' + ); + + const result = await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'square' }, + 'android', + 'none' + ); + + expect(result).toEqual({ pass: true, message: 'Images match' }); + expect( + existsSync(path.join(path.dirname(snapshotPath), 'square-actual.png')) + ).toBe(false); + expect( + existsSync(path.join(path.dirname(snapshotPath), 'square-diff.png')) + ).toBe(false); + }); + + it('saves the diff and received images of a mismatch beside the snapshot', async () => { + await mkdir(path.dirname(snapshotPath), { recursive: true }); + await writeFile(snapshotPath, createPng({ fill: 0 })); + + const result = await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'square' }, + 'android', + 'none' + ); + + expect(result.pass).toBe(false); + expect( + await readFile(path.join(path.dirname(snapshotPath), 'square-actual.png')) + ).toEqual(await readFile(receivedPath)); + expect( + existsSync(path.join(path.dirname(snapshotPath), 'square-diff.png')) + ).toBe(true); + }); + + it('names the received image of a missing snapshot after the whole snapshot name', async () => { + await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'logo.png-dark' }, + 'android', + 'none' + ); + + expect( + existsSync( + path.join(path.dirname(snapshotPath), 'logo.png-dark-actual.png') + ) + ).toBe(true); + }); + + it('names the images of a mismatch after the whole snapshot name', async () => { + await mkdir(path.dirname(snapshotPath), { recursive: true }); + await writeFile( + path.join(path.dirname(snapshotPath), 'logo.png-dark.png'), + createPng({ fill: 0 }) + ); + + await matchImageSnapshot( + { path: receivedPath }, + testFilePath, + { name: 'logo.png-dark' }, + 'android', + 'none' + ); + + expect( + existsSync( + path.join(path.dirname(snapshotPath), 'logo.png-dark-actual.png') + ) + ).toBe(true); + expect( + existsSync( + path.join(path.dirname(snapshotPath), 'logo.png-dark-diff.png') + ) + ).toBe(true); + }); +}); diff --git a/packages/bridge/src/image-snapshot.ts b/packages/bridge/src/image-snapshot.ts index 332d40f6..eabff150 100644 --- a/packages/bridge/src/image-snapshot.ts +++ b/packages/bridge/src/image-snapshot.ts @@ -3,7 +3,11 @@ import { ssim } from 'ssim.js'; import fs from 'node:fs/promises'; import path from 'node:path'; import { PNG } from 'pngjs'; -import type { FileReference, ImageSnapshotOptions } from './shared.js'; +import type { + FileReference, + ImageSnapshotOptions, + SnapshotUpdateState, +} from './shared.js'; type PixelmatchOptions = Parameters[5]; @@ -49,7 +53,8 @@ export const matchImageSnapshot = async ( screenshot: FileReference, testFilePath: string, options: ImageSnapshotOptions, - platformName: string + platformName: string, + updateSnapshot: SnapshotUpdateState ) => { const pixelmatchOptions = { ...DEFAULT_OPTIONS_FOR_PIXELMATCH, @@ -69,14 +74,24 @@ export const matchImageSnapshot = async ( const testDir = path.dirname(testFilePath); const snapshotsDir = path.join(testDir, SNAPSHOT_DIR_NAME, platformName); - const snapshotName = `${options.name}.png`; - const snapshotPath = path.join(snapshotsDir, snapshotName); + const snapshotPath = path.join(snapshotsDir, `${options.name}.png`); + const failureImagePath = (image: 'actual' | 'diff'): string => + path.join(snapshotsDir, `${options.name}-${image}.png`); + const actualPath = failureImagePath('actual'); await fs.mkdir(snapshotsDir, { recursive: true }); try { await fs.access(snapshotPath); } catch { + if (updateSnapshot === 'none') { + await fs.writeFile(actualPath, receivedBuffer); + return { + pass: false, + message: `New snapshot was not written: ${snapshotPath} does not exist, and missing snapshots are not written in CI mode. Received image saved at ${actualPath}; commit it as ${snapshotPath}, or run without CI mode (--ci=false) on a machine with this device and commit the snapshot it writes.`, + }; + } + // First time - create snapshot await fs.writeFile(snapshotPath, receivedBuffer); return { @@ -152,12 +167,9 @@ export const matchImageSnapshot = async ( // Save diff and actual images when test fails if (!pass) { - const diffFileName = `${snapshotName.replace('.png', '')}-diff.png`; - const diffPath = path.join(snapshotsDir, diffFileName); + const diffPath = failureImagePath('diff'); await fs.writeFile(diffPath, PNG.sync.write(diff)); - const actualFileName = `${snapshotName.replace('.png', '')}-actual.png`; - const actualPath = path.join(snapshotsDir, actualFileName); await fs.writeFile(actualPath, receivedBuffer); message += ` Diff saved at ${diffPath}. Actual image saved at ${actualPath}.`; diff --git a/packages/bridge/src/server.ts b/packages/bridge/src/server.ts index 8b452164..38f884c6 100644 --- a/packages/bridge/src/server.ts +++ b/packages/bridge/src/server.ts @@ -166,7 +166,13 @@ export const createHarnessBridge = async ( localMethods: { 'device.screenshot.receive': (ref) => receiveScreenshot(link, ref), 'test.matchImageSnapshot': (screenshot, testPath, opts) => - matchImageSnapshot(screenshot, testPath, opts, context.platform.name), + matchImageSnapshot( + screenshot, + testPath, + opts, + context.platform.name, + context.updateSnapshot ?? 'new' + ), }, transport: { send: (message) => { diff --git a/packages/bridge/src/shared.ts b/packages/bridge/src/shared.ts index 3cfa22d5..bbd28e40 100644 --- a/packages/bridge/src/shared.ts +++ b/packages/bridge/src/shared.ts @@ -169,6 +169,10 @@ export type BridgeServerFunctions = { ) => Promise<{ pass: boolean; message: string }>; }; +export type SnapshotUpdateState = 'all' | 'new' | 'none'; + export type HarnessContext = { platform: HarnessPlatform; + /** Jest's snapshot update state; when absent, a missing image snapshot is written. */ + updateSnapshot?: SnapshotUpdateState; }; diff --git a/packages/cli/skills/ui.md b/packages/cli/skills/ui.md index 34d9bf97..b5e1e5d2 100644 --- a/packages/cli/skills/ui.md +++ b/packages/cli/skills/ui.md @@ -37,7 +37,8 @@ Use this skill when the task requires: - Keep imports split correctly: core APIs from `react-native-harness`, UI APIs from `@react-native-harness/ui`. - Mention that `@react-native-harness/ui` requires installation, and native apps must be rebuilt after adding it. -- `toMatchImageSnapshot(...)` needs a unique snapshot `name`. +- `toMatchImageSnapshot(...)` needs a unique snapshot `name`, and cannot be used with `.not`. +- In Jest's CI mode, on by default in CI, a missing snapshot fails and its received image is saved as `-actual.png`: commit that image as `.png`, or run with `--ci=false` on the device that compares it and commit the snapshot it writes. - If screenshotting elements that extend beyond screen bounds, call out `disableViewFlattening: true` in `rn-harness.config.mjs`. - On web, UI interactions and screenshots run through the web runner's Playwright-backed browser environment. diff --git a/packages/jest/src/__tests__/bridge.test.ts b/packages/jest/src/__tests__/bridge.test.ts index b609ae7c..38abab6d 100644 --- a/packages/jest/src/__tests__/bridge.test.ts +++ b/packages/jest/src/__tests__/bridge.test.ts @@ -3,7 +3,10 @@ * connectToHarness (app side). Tests the full connection lifecycle and * RPC round-trip without coupling tests to the transport internals. */ +import { existsSync } from 'node:fs'; import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { HarnessBridge } from '@react-native-harness/bridge/server'; import { createHarnessBridge } from '@react-native-harness/bridge/server'; @@ -11,13 +14,16 @@ import { connectToHarness } from '@react-native-harness/bridge/client'; import type { HarnessContext } from '@react-native-harness/bridge'; import type { TestSuiteResult } from '@react-native-harness/bridge'; -const makeContext = (): HarnessContext => ({ +const makeContext = ( + updateSnapshot: HarnessContext['updateSnapshot'] = 'new', +): HarnessContext => ({ platform: { name: 'ios', platformId: 'ios', runner: '/dev/null', config: {}, }, + updateSnapshot, }); // --------------------------------------------------------------------------- @@ -236,6 +242,64 @@ describe('bridge: createHarnessBridge + connectToHarness', () => { }); }); + describe('image snapshots', () => { + it.each([ + [ + 'writes a missing snapshot and passes when the run passes updateSnapshot new', + makeContext('new'), + true, + ], + [ + 'fails a missing snapshot without writing it in CI mode when the run passes updateSnapshot none', + makeContext('none'), + false, + ], + [ + 'writes a missing snapshot and passes when the context names no snapshot state', + { platform: makeContext().platform }, + true, + ], + ] as const)( + '%s', + async (_, context, written) => { + bridge.dispose(); + bridge = await createHarnessBridge({ port: 0, context }); + bridgePort = (bridge.ws.address() as { port: number }).port; + + const directory = await fs.mkdtemp( + path.join(os.tmpdir(), 'harness-bridge-snapshot-'), + ); + + try { + const receivedPath = path.join(directory, 'received.png'); + await fs.writeFile(receivedPath, 'received image'); + + const connectionPromise = bridge.nextConnection(); + const handle = await connect(); + handle.reportReady(device); + await connectionPromise; + + const result = await handle.matchImageSnapshot( + { path: receivedPath }, + path.join(directory, 'example.harness.tsx'), + { name: 'square' }, + 'ios', + ); + + expect(result.pass).toBe(written); + expect( + existsSync( + path.join(directory, '__image_snapshots__', 'ios', 'square.png'), + ), + ).toBe(written); + handle.disconnect(); + } finally { + await fs.rm(directory, { force: true, recursive: true }); + } + }, + ); + }); + describe('dispose', () => { it('rejects pending nextConnection() waiters', async () => { const pending = bridge.nextConnection(); diff --git a/packages/jest/src/harness-session.ts b/packages/jest/src/harness-session.ts index c23ee613..0e211c55 100644 --- a/packages/jest/src/harness-session.ts +++ b/packages/jest/src/harness-session.ts @@ -622,7 +622,10 @@ export const createHarnessSession = async ( const getCurrentRunId = () => currentRun?.runId; const clientLogCollector = createClientLogCollector(); - const context: HarnessContext = { platform }; + const context = { + platform, + updateSnapshot: globalConfig.updateSnapshot, + } satisfies Required; const crashArtifactWriter = createCrashArtifactWriter({ runnerName: platform.name, platformId: platform.platformId, diff --git a/packages/runtime/src/__tests__/to-match-image-snapshot.test.ts b/packages/runtime/src/__tests__/to-match-image-snapshot.test.ts new file mode 100644 index 00000000..337b2511 --- /dev/null +++ b/packages/runtime/src/__tests__/to-match-image-snapshot.test.ts @@ -0,0 +1,62 @@ +import { GLOBAL_EXPECT } from '@vitest/expect'; +import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import '../globals.js'; + +const transferScreenshot = vi.fn(async () => ({ + path: '/project/received.png', +})); +const matchImageSnapshot = vi.fn(); + +vi.mock('../client/store.js', () => ({ + getHandle: () => ({ transferScreenshot, matchImageSnapshot }), +})); + +vi.mock('../runner/index.js', () => ({ + getHarnessContext: () => ({ + testFilePath: '/project/screen.harness.tsx', + runner: 'android', + }), +})); + +// The runtime's expect module installs itself as the global expect; Vitest's +// own expect is put back once the tests are done. +const vitestGlobalExpect = Object.getOwnPropertyDescriptor( + globalThis, + GLOBAL_EXPECT +); +const { expect: harnessExpect } = await import('../expect/expect.js'); + +afterAll(() => { + if (vitestGlobalExpect) { + Object.defineProperty(globalThis, GLOBAL_EXPECT, vitestGlobalExpect); + } +}); + +const screenshot = { data: new Uint8Array(4), width: 1, height: 1 }; + +describe('toMatchImageSnapshot', () => { + beforeEach(() => { + transferScreenshot.mockClear(); + matchImageSnapshot.mockReset(); + }); + + it('fails with the message of a comparison that did not pass', async () => { + matchImageSnapshot.mockResolvedValueOnce({ + pass: false, + message: 'New snapshot was not written', + }); + + await expect( + harnessExpect(screenshot).toMatchImageSnapshot({ name: 'square' }) + ).rejects.toThrow('New snapshot was not written'); + }); + + it('refuses .not before a snapshot is compared or written', async () => { + await expect( + harnessExpect(screenshot).not.toMatchImageSnapshot({ name: 'square' }) + ).rejects.toThrow('`.not` cannot be used with `.toMatchImageSnapshot()`.'); + + expect(transferScreenshot).not.toHaveBeenCalled(); + expect(matchImageSnapshot).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/runtime/src/expect/matchers/toMatchImageSnapshot.ts b/packages/runtime/src/expect/matchers/toMatchImageSnapshot.ts index 33779f9a..1f4b3cb2 100644 --- a/packages/runtime/src/expect/matchers/toMatchImageSnapshot.ts +++ b/packages/runtime/src/expect/matchers/toMatchImageSnapshot.ts @@ -14,6 +14,10 @@ export async function toMatchImageSnapshot( received: ScreenshotResult, options: ImageSnapshotOptions ): Promise<{ pass: boolean; message: () => string }> { + if (this.isNot) { + throw new Error('`.not` cannot be used with `.toMatchImageSnapshot()`.'); + } + const handle = getHandle(); const context = getHarnessContext(); diff --git a/packages/runtime/src/globals.ts b/packages/runtime/src/globals.ts index d3a63971..7ddaf1a3 100644 --- a/packages/runtime/src/globals.ts +++ b/packages/runtime/src/globals.ts @@ -13,7 +13,8 @@ declare module '@vitest/expect' { interface Matchers { /** * Match the received screenshot against a stored snapshot. - * Creates a new snapshot if one doesn't exist. + * Creates a new snapshot if one doesn't exist, except in Jest's CI mode, + * where a missing snapshot fails the test. Cannot be used with `.not`. */ toMatchImageSnapshot(options: ImageSnapshotOptions): Promise; } diff --git a/website/src/docs/api/cli.mdx b/website/src/docs/api/cli.mdx index 940e2d4d..93f66d17 100644 --- a/website/src/docs/api/cli.mdx +++ b/website/src/docs/api/cli.mdx @@ -76,6 +76,7 @@ React Native Harness supports the most commonly used Jest flags: - `--testNamePattern, -t`: Run only tests with a name that matches the regex. - `--testPathPattern`: A regexp string that is matched against all tests paths before executing the test. - `--verbose`: Display individual test results with the test suite hierarchy. +- `--ci`: Run in CI mode, where a missing image snapshot fails the test instead of being written. Jest turns it on in a CI environment it detects; `--ci=false` turns it off. ## Restrictions and Limitations @@ -107,4 +108,4 @@ The following Jest configuration flags are **disabled**: ### Snapshot Management -Snapshot testing for UI is handled via the separate `@react-native-harness/ui` package and the `toMatchImageSnapshot` matcher. Standard Jest snapshot flags like `--updateSnapshot` (`-u`) are currently not supported for native image snapshots. +Snapshot testing for UI is handled via the separate `@react-native-harness/ui` package and the `toMatchImageSnapshot` matcher. Standard Jest snapshot flags like `--updateSnapshot` (`-u`) are currently not supported for native image snapshots. A missing snapshot is written, except in CI mode (`--ci`), where the test fails instead; see [Creating snapshots](/docs/guides/ui-testing#creating-snapshots). diff --git a/website/src/docs/guides/ui-testing.mdx b/website/src/docs/guides/ui-testing.mdx index 0705d992..61860e87 100644 --- a/website/src/docs/guides/ui-testing.mdx +++ b/website/src/docs/guides/ui-testing.mdx @@ -148,6 +148,14 @@ test('screenshot with custom options', async () => { | `diffColor` | `[number, number, number]` | `[255, 0, 0]` | RGB color for highlighting differences | | `ignoreRegions` | `Array<{x,y,width,height}>` | `[]` | Regions to exclude from comparison | +### Creating snapshots + +`toMatchImageSnapshot` stores each snapshot as `__image_snapshots__//.png` next to the test file. When that file does not exist, the test writes it and passes, except in Jest's [CI mode](https://jestjs.io/docs/cli#--ci), where the test fails without writing it and saves the received image beside it as `-actual.png`. Jest turns CI mode on when it detects a CI environment, such as one with the `CI` environment variable set; `--ci` and `--ci=false` override that. + +Snapshots are compared pixel by pixel, so create them on the device, simulator or browser that compares them: commit the `-actual.png` a CI run saved for a missing snapshot as `.png`, or run with `--ci=false` on a machine with that device and commit what it writes. The [GitHub Action](/docs/guides/ci-cd) uploads the `-actual.png` images of a run as its `visual-test-diffs-` artifact; in other CI, upload `**/__image_snapshots__/**/*-actual.png` from the job. + +Like Jest's own snapshot matchers, `toMatchImageSnapshot` cannot be negated: `expect(screenshot).not.toMatchImageSnapshot(...)` throws. + ### Element-specific screenshots You can capture screenshots of specific elements instead of the entire screen: