j udZddlmZddlZddlZddlmcmZddl m Z m Z m Z m Z ddlmZmZdZdZd Zd Zd Zd ZdvdZdwdddvdZdxdddydZdzddd{d!Zd|d&Zd}d(Zd~d*Zdd2Zddd4d5d6dd7Zdd9Z dd>Z!dddBZ"ddDZ#ddFdGddIZ$ddJZ%ddLddFdMddOZ&dPdQdRdSdTdUedVZ'dWddXdYd?dZdd^Z(d~d_Z)ddWddXddFdd`ddaZ*db\Z+Z,dcZ-ddZ.ddhZ/ddjZ0dddmZ1ddddndoddsZ2ddWddXdddtdduZ3dS)aCameras, multi-mode rendering, previews and turntables, built on raster.py. Cameras follow OpenGL: look_at() returns world-to-camera (view) matrices, the camera looking down its -Z axis with +Y up and +X right; perspective() / orthographic() return projection matrices to clip space (z / w = -1 at the near plane, +1 at the far plane). A camera set is a (view, projection) pair, each [4, 4] or [B, 4, 4]. Rendered images are channels last, top row first: [B, H, W, C]. Colours are sRGB in 0..1; shading is computed in linear light. ) annotationsN)Meshlinear_to_srgbsrgb_to_linearvertex_normals) interpolate rasterize)shadedalbedonormaldepthpositionmask)r r gsR?)gܿg?g333333?)gA@g4@g>@return torch.Tensorct|tjr!||tjn tj|tj|S)N)devicedtyperr) isinstancetorchTensortofloat32tensor)valuers zlook_at..#s-EEawq&))EEEEEEr dimrrN.N.rr)rrrrndimreshapeF normalize expand_aslinalgcrossstackeyerrepeatlen) r>targetuprbatchedforwardrightrotationviews ` rlook_atrHsr&c5<88 DSZZfFEEEEC3DEEEOCh!mG ++b!  Ck&**3//#52>>>G K **7BLL4E4EFFB O O OE{E5<#5#5eW#E#ExPRSTTH 9QemF ; ; ; B B3s88QPQ R RDDBQBO#i.0&99DBQBN '44Q'r D@r$皙?Y@fov_yfloataspectnearfarc dtjtj|dz z }tj||z dddgd|ddgdd||z||z z d|z|z||z z ggdgtj|S)zaOpenGL perspective projection [4, 4]: vertical field of view in degrees, aspect = width / height.r$r.r)rrr/rr)mathtanradiansrrr)rLrNrOrPrfs r perspectiverV/s dht|E**Q.///A  linear float [1, 3, h, w], area-downsampled by an integer factor to at most twice the render size (a cheap stand-in for mipmapping), converted by bands of rows to bound memory.rrr.@r0c g|]g}tj|t|zdddfdddddz hS)Nr0r.rr)r8 avg_pool2dminpermuterM)r+rbandfactorrXr\widths r z_texture..Bs V V VyzQ\'!CD&$9$9"96E62A2"EFNNqRSUVWWX\]cceehkkms t t V V Vr )maxrRceilshaperangerrcat)r\r]r_bandsrhrirXrjs` @@@@r_texturerr<s(DIc'-"2a*n"Ew}UVGW[\_h[hGijjkk l lFM!$.7q9IV9SV\9\MFE CGV);<== =D V V V V V V V VDEFHNPTUU V V VE %)E1-- . ..r uvsc||z dzdz dddd}tj||ddddd d dfjS) zUBilinear lookup (repeat wrap) of a [1, C, h, w] texture at glTF uvs [N, 2] -> [N, C].r.rr/bilinearborderF)mode padding_mode align_cornersrN)floorrGr8 grid_sampleT)r\rsgrids r_sampler~Fsf 399;; ! #a ' - -aB : :D =$Zh^c d d defhihihiklel m oor !tuple[torch.Tensor, torch.Tensor]c2fd|D\}}|ddd|ddd}}tt|t|}||dd||ddfS)Nc38K|]}t|VdSr)r*)r+mrs rr-z_cameras..Ms-<bhwir rctj|ddddddddS)Nrr0rr.)r8rdrf)xrs rz_render..s=aii1a&;&;[IIQQRSUVXY[\]]r )%MODES ValueErrorverticesrrrr^rrprM ones_liker transposeneg_r facesr normalsrrs base_color COLOR_MODEScolorsrr8r9einsum full_likeDEFAULT_ALBEDO_LINEARr~rrwhererLIGHT clamp_minrclamp)!rr resolutionrwrrrrrGrrXrjhw homogeneousrclipface_idbary_coveredvaluesrtextured attributespixelsr r hitcolor normal_cameralightpools! ` r_renderrSs> 5888999 ] !F00D*0::s0K0K[Z,,Q[MFE K !4qA)T]0022EODMRSRSRSUWVWUWRWDX`e`m4n4n4noqrssK 4>>!Q// /F J((A.. .DL tzAq6::GT1!|Y'--//G v~~   T]0022DJNN fS!A#X. GTJJJ?C|?S. ;;;Y]YeYkYkYmYm84'GDO4,GY ;   @48>>++,,b @J  3""%)^TXT_`a`a`acedece`eTfTlTlTnTnEoEoquq|}~}~}~ABACAC~CrDrJrJrLrLEMOP;Q;Q:R RJUYz155tz7DQQVC!G_"555 8  P\`hPhPhU\"2DBQBOVLLLntFF_VC!G_6KLLF ^l%ht1&E&EvcSTUVSVhGWX[G\]]s {&s$8AAq$9$99:37C+3>%x % -=tAAArr2A2IPV W W % M#qs(,Ca,G-Yf g g  ELemTZ$[$[$[abccc7a'kme>S=^=^_`=a=abk=l-l#lm#FLLA$6$677F g FQ]]]]$v,,W  7?r r world333333?)rrrc t|||||||\}} |8|tvr-tj|| dz | gdn|S|t ||jd| z zzS)aRenders B views of a mesh: [B, H, W, C] float32 on the mesh's device (resolution: int or (H, W)). mode: - "shaded": sRGB colour, Lambert key light from the camera's upper left plus `ambient`, two-sided; - "albedo": unlit sRGB colour (base colour texture x vertex colours, or grey 0.8); - "normal": unit normals (C = 3) in world space, or camera space (x right, y up, z towards the camera) with normal_space="camera"; normal maps are ignored; - "depth": distance along the camera axis (C = 1); - "position": world positions (C = 3); - "mask": coverage (C = 1). With background None, colour modes return RGBA (straight alpha, C = 4) and the other modes are zero where empty; otherwise every mode is composited over background (a value or per-channel sequence). supersample renders at that factor and averages down (anti-aliasing, fractional coverage on edges).N-q=r/r)rrrrprrr) rrrrw backgroundrrrrrs rrenderrsdGZ{LZabbOFGNRVaNaNauy&7#4#4U#;#;;WErJJJgmm GJ 66!g+F FFr tuple[torch.Tensor, float]c|jd|jd}}||zdz }|t t|j|z ddfS)Nrr.rr1ư>)rrMaminamaxrlnorm)rlowhighcenters r_framingrs ##%%**1--t}/B/B/D/D/I/I!/L/LCDjA F 3udm1133f<BBqBIIMMOOPPRVWW WWr rradiusazimuth elevationc r|tjtjtdz z }tj|tj|}}t jtj|tj|ztj|tj|tj|zgtj|j}t|||zz|}ttdt|d|zz d|z|d|zz|j} || |fS)ziView and projection of a camera orbiting center at the distance where the bounding sphere fills the view.r.rr$?MbP?r&) rRsinrT FIELD_OF_VIEWrrcosrrrHrVrl) rrrrdistanceae directionrGrs r_orbitrs'm!rr)rrrrrhalfscaleadjusts r_zoomrs$J!#UTCZ1,<+A+A+C+C+M+Md+S+S%T%TDF 4KE Yq j6G H H HF"''F4L6$<"'&)!3eVfQi5GF4L&, J r rGc dtj|jtj|jddddftjgd}|||zjz}|ddddf|ddddfz }|d|dfS)Nrrr.r0r) rrprrMrrr|rr)rrGrrrndcs r_ndc_boxrs)T]0022EODMRSRSRSUWVWUWRWDX`e`m4n4n4noqrssK *t+. .D qqq"1"u+QQQU #C 88A;; ##r r.rsizect|\}}t||gtR\}}}t|gt |||R}t |||f|d|d}|ddddzdz S)zCanonical 3/4 view (azimuth 35 degrees from the front towards +X, elevation 20 degrees), shaded, auto-framed on the mesh: RGBA [4, size, size] in [-1, 1] with a transparent background (alpha -1), on the mesh's device.r rrr.r)rr PREVIEW_VIEWrrrrf) rrrrrrGrrimages rpreviewrsd^^NFF ?,???D*azEHT4$D$DEEEJ 4$ +T8 U U UVW XE ==Aq ! !A % ))r c  |tvrtd||jj}t |t r||fn|\}} |duo|t v} | rdnt|dn||}t|\} } t|} tdd|| z|z|zz}tj | || | rdndtj |}td | |D]}t||z| }t!|||||||f|| f||d d \}}|d kr"|d |d c|||dd f<|||ddf<i| r3tj||dz |gd|||<|dkr ||zdz }n|dkr|| | z |zz d| zz }||d|z zzdddd|||<|d kr0|d}|d |dz }||dk}t|r(|| fn+tjd|tjd|f\}}|dd||z z||z d| zz z d dz}|d|ddddd|dz zzdddd|dd<|S)aViews [B, H, W, C] in 0..1 rendered by groups (bounded memory). Colour modes: RGBA with straight alpha when background is None (C = 4), else sRGB over background. Other modes (C = 3, over black by default): normal: camera-space (n + 1) / 2; depth: linear from 1 at the nearest to 0.15 at the farthest surface seen in all the views (no flicker between views); mask: coverage; position: world coordinates mapped from the bounding sphere to 0..1.rNr"rrar3r0rrrrrr5.rr/r r.rrg+?r&333333?rr4)rrrrrr^rrrr@rlremptyrrorerrprrclonezerosonesrr7)rrGrrrwrrrrXrj transparentrrrgroupimagesstartstoprcoveragersolidrOrPs r_encoded_viewsrs  5888999 ] !F0::s0K0K[Z,,Q[MFE$<)''))v!3!3E!:!::h&'25e**BUYY[[%))++..5;rZ`CaCaCachcmnpydAdAdACB cQ!6#*9O9OPTW]P]9^9^!^^eefgijkkkI&););Aq!R)H)HAPXYbPcLc)ddllmoqsuwyz{{qqq Mr 0g.@)rrrframesc  |tvrtd|t\  fdtD}fd|D}t jd|Ddt jd|Dd} } t j| | t jd|Dt jfd|D} } ||n |tvrd nd }t| | |||| d dd ddzd z S)a/Turntable video [3, frames, size, size] in [-1, 1]: the camera orbits the mesh (starting at the front, +Z, moving towards +X) with the key light attached to it, framed so that the mesh stays in view on every frame. Encodings as _encoded_views; shaded / albedo over a white background by default.rc@g|]}td|zz S)gv@)r)r+irrrrs rrkzturntable..s0 [ [ [fVVUQY%7CC [ [ [r c:g|]\}}}t||Sr)r)r+rGrrrs rrkzturntable..s, P P P2E$ AXdD* - - P P Pr cg|] }|d S)rrr+bs rrkzturntable..s111aQqT111r rcg|] }|d S)rrrs rrkzturntable..sH]H]H]RS1H]H]H]r cg|]\}}}| Srr)r+r,rs rrkzturntable..s#<#<#<'!QA#<#<#.s5KxKxKxfmfgijlmERSVZUZ\`LaLaKxKxKxr N)r$r$r$r"r0rr.) rrrrorr=rrmaximumabsrrrf)rrrrwrrrorbitsboxesrrrGrrrrs`` ` @@@r turntabler s  5888999d^^NFF [ [ [ [ [ [ [USY]] [ [ [F P P P P P P PE 1151112277::EKH]H]W\H]H]H]<^<^ 0, against splat floaters), magnified by zoom, centred on focus (fractions of the bounding box: 0.5 is the centre, y 1 the top, z 1 the front).Nr rr1r.rrrrVr&rr rr$gyV@r#r"r0r3rr)#rr^rrlr@rMrrrrrrrRrrTrrVr[rprrrerrrcopysignr rHr|rappendr>rr=r)"ranglesrrrrrrrXrjrsamplerrrrrArrOrPbaserviewsrrrrrrsignrBrrrs" r orbit_camerasr#s1;:s0K0K[Z,,Q[MFE ]F 33c!S[[F2333 4 : : < >4RX=@X#DS]anSnSn;}efndC O O O OuABCFLBLNSV\N\^bdgpvuwuwuwD)VU_VAAArrE]%C%CDaHHKek!F3334E$II|G$$dl3uc$ >R>R3S3S&T&T1L$(1++ ";TXa[[$(ST++X\X`abXcXcJc!dlqlyCIJJJ }S),, #I 5 5__TXa[[L4#q8|#K#K#K#Kadaiaijkalalmm WVi(&::FBGGHHHH Yq f = = =F"&+dhhjj6J6J46P6P0Q0Q"QQF4L6$< ;u   55c%jj!QGG GGr c t|trt|j||fi|St|d||fddd|S)zorbit_cameras of a mesh, or of Gaussian splats framed on the 2-98 % quantiles of their centres (against floaters): the cameras render_views / render_splat_views use, computed once for a sequence rendered in slices.means{Gz?r)rr)rrr#r)assetrrframings r asset_camerasr)%sZ%LU^VZKK7KKK w adQU a aY` a aar )rrrrrrc d| t||||||n| \} }t|| |||||S)aViews [B, H, W, C] in 0..1 of a mesh from orbit_cameras (one scale shared by every view), encoded as _encoded_views: RGBA with a transparent background for the colour modes by default. cameras: a slice of the asset_cameras of a longer sequence, rendered instead of angles.Nrrr)r)r) rrrrwrrrrrrrGs r render_viewsr,-sU ryrA}T6:*[_glmmmmGND* $j*dJP[ \ \\r )gmBP ?gtWxCE?)g+{?+{gg`/?r-g+{?);a>g-\@V^=@ݿgh?r/g-\?r.splatsdict directionsc|dg|dgn9t|ddz}d|dD\}}}t|dz}t |dkr>|t |z|dzz t |z|dzzt |z|dzz }t |d kr||z||z||z} }}|t d|z|z|d zzt d|z|z|d zzt dd| z|z |z z|d zzt d|z|z|d zzt d ||z z|d zz}t |dkr|td|zd|z|z z|dzztd|z|z|z|dzztd|zd | z|z |z z|dzztd|zd| zd|zz d|zz z|dzztd |zd | z|z |z z|dzztd |z||z z|dzztd |z|d|zz z|dzz}|dzdS)zColours [N, 3] of splats seen along unit directions [N, 3] (from the camera to each splat), evaluating their spherical harmonics (degrees 0 to 3) as 3D Gaussian Splatting does: 0.5 + sum of the SH terms, clamped at 0.sh_dcsh_restNrc3.K|]}|dddfVdSr)r)r+ds rr-zsplat_colors..@s.88aqDz888888r rr.r0r3r r) rMlistunbindSH_C0r@SH_C1SH_C2SH_C3r) r0r2shryzresultxxyyzzs r splat_colorsrO<sf /   ! ! "F9,=,Ebb4PVW`PaPgPgPiPiPpPpqrPsPsKtKt uB88:#4#4Q#7#7888GAq! RU]F 2ww{{%!)be++eai"Q%.??%!)bQReBSS 2ww{{UAE1q5B58a#xQ#G#G#GX[]^ai]iopIqIqIqx|}}} }}r i)r max_radiusfragment_budgetrU int | NonerVc |d}|j}t|tr||fn|\}}|t |ndt ||D\} } |p t dt||dz}tj |tj |ddddfgd| j z} | | j z} | dddf} tj | dddf| z dz|zdz d z d| dddf| z z |zdz d z gd}| dddddf| ddddfz| dddddf| dddddfzz | | zddddfz }|tj |dz | dz g| ddddfz| dddddfz}|t|d |d dddddfzz}||ddzd tjd| zz}|ddddf|ddddf|ddddf}}}||z||zz }tj || |gd|dddfz }tjd||zdz ||z dz dz||zzzzd|}| dddf dk| dkz|ddkz|dkz}|||dddfzdk||dddfz tj |dz |dz g| kzdz}|d}|tj| |df d}| ddddfj | dddfz}|ddn |d|}t3|d||dt5j|||z d}|||||||d|f\}}}}tj||zdtj|}tj||ztj|}d|zdzdz} | d}!d}"|"tA|krt ttj!|!|!|"| |"z |zd|"dz}#| |"|#}$tj"tj#|"|#| |$}%tj#tA|%| tj"|$d|$z |$z }&d||%zdz}'tj$||%df|&%|'dz||%z }(tj$||%df|&|'zz||%z })|)||%dfz |(||%dfz }+}*d||%df|*z|*z||%df|+z|+zzz||%df|*z|+zz },||%|,&z'd}-|-dk|)dkz|)|kz|(dkz|(|kz}.|(|z|)z|.|-|.(|%|.}%}-}/tj)|/d\}/}0|-|0|%|0}%}-tj*|- }1|1d|1z }2tj |/tj+}3|/dd|/ddk|3dd<|2|2tj,tj-|3tj#tA|/| ddj.z }2||/|2z&|-z}4|/d|/|4dddf||%z|/d|/|1|#}"|"tA|kd|&z }5||50ddddfz dd}6tj |6|5dddfgdj 1d||dzdz S)aThumbnail of Gaussian splats (see splats.py for the dict): RGBA [4, H, W] in [-1, 1] (size: int or (H, W)), transparent background, on the device of the splats. Pure PyTorch (no CUDA extension): each splat is projected as an elliptical 2D Gaussian (3DGS EWA projection with the 0.3 pixel low-pass), coloured by its spherical harmonics for the view direction, and alpha-blended front to back in depth order (one depth per splat, as 3DGS). cameras: (view, projection) like render(); by default the canonical 3/4 view framed on the 2-98 % quantiles of the centres. Splat footprints are clipped to max_radius pixels (a quarter of the smaller side by default); fragment_budget bounds the memory (about 190 bytes per fragment).r%Nc3&K|] }|dV dS)rNr)r+rs rr-z splat_preview..ls&EnEnqadEnEnEnEnEnEnr r.r3rr0rrr& rotationsscalesr opacitiesgp?T)stabler5r4)r4r5r1r)rErz) rounding_modeggGz?rr/r)2rMrrr^rTrrlrerrprr|r=rrRrr>rmsqrtrlongallnonzerosqueezeargsortrOr8r9rrfloat64cumsumr@ searchsortedrepeat_interleavearangerounddivexp clamp_maxdoublesortlog1pboolcummaxrr index_add_rr7)7r0rrrUrVr%rrXrjrGrrrrcentrejacobian footprint covariancerrc determinantconicrvisibleorderr>restropacityrlog_transmittancecountsendsrrspansplatlocalsiderowcoloxoypoweralphakeeppixelro attenuationbeforefirstweightrrgbs7 r splat_previewras$  7O ! ! # #E \F$.tS$9$9CT4LLtMFE07~e,,,EnEnT\]dflTmTmEnEnEnD*>s1c&%&8&8A&=>>J YuuQQQU|<<=q A ADF JF JL D QQQT A [41:>A-6:S@1tAAAqDzTU~CUY_B_bcBcfiBijlm n nF4!RaR<(1QQQd]+;;d111bqb$;>OR\]acdcecegihigi]iRj>jjopstotvwvwvwy}@DwDoEEH5<VGaK(@PPPQUWXWXWXZ^Q^__cghlnpopnprtstrthtcuuH0 1DEExHXH^H^H`H`abababdhjkjkjkakHllmIY00A666uySY?Z?Z?Z9ZZJAq!:aaaAg#6 111a78K!qAa%!a%-K KQB A & &QQQW)= =E Za!eq[a!eq[Q,>Q,F+L+L+N+NNTTVVV W W ] ]^_ak l l q q s sFqqq!t }q QU+vk/BW/LMQ\_`Q`aG &D/)Q.6F111d7O3Ku|]bef]fhnqrhr\s}CPDPDPD4DEJJKLMMMG OO   % %a ( (E %-q!1 1$??? @E BQB<>/D!QK 'C)$,44&2CE2JD F7OE$:tLLakZ_`eZfilZlrsNtNtNt u uF%+E]E%L&-QWXcQdejQkQqQqQsQs%s"FE67 Kv N N NE FUN%-PVWWW&j1n "F ==  D E #e**  3u)$U fUm0Ko0]eijjjkkmruvmvwweDj!' UD(P(P(PRVWW SZZ777%:QRVR]R]^_R`R`cgRgim:n:nn6%= 1$k&*++0022UYYtSZY5[5[[^dej^kkk&*++0022UT\AF5MQveQh''veQh/?)?BeQh",r1E%(Ob4H24MMNQVW\^_W_Q`ceQehjQjj%))++-88>> SAX.#+>#(KsU[|\"U{S0$7t9K9K9M9MuUY{eujt444 tT{E$Kuk5&)) ##A&&4UZ888!""Iss+abb &ek%cRWjjagAhAhAhjk.l.lno!p!p!wxx$U+f499;;eCJJLL E6!!!T'?VE]#BCCC$$Q{;;;/ #e**  0%))+++2244H 8%%e,,QQQW5 5 < .sjOOOptpqst=&%1a&QQQYYZ[]^`abbeffjkkOOOr .r0rr/r) rrr^r)rr=ziprrr)r0rrrwrrrrrrGrrrrXrjs` @@rrender_splat_viewsrsc  ///_W[__```0::s0K0K[Z,,Q[MFExyH}VVfe_Q[bfnsttttNUD* [OOOOOOx{}ACMyNyNOOOPPF37OVCG_C v~~r2r1---'66S8^gjRUR\>]>]abemam>n-nnr )rr)r!r#)rIr$rJrK) rLrMrNrMrOrMrPrMrr)rWr$rJrK) rXrMrNrMrOrMrPrMrr)r\rr]r^r_r^rr)r\rrsrrr)rr) rrrwrrr^rrrrM)r N) rrrwrrr^rrrrMrr)rrrr)rrrrMrrMrrM)r)rrrrMrr)rrrGrrr)r)rrrr^rr^rr) rrrGrrrrwrrr^rr)rrr )rrrr^rr^rwrrrMrr^rr) rrrrrrMrrMrrMrr)r ) rrrwrrrrrMrr^rr)r0r1r2rrr)rPrrr)r&)r%rrrM)r0r1rUrWrVr^rr) r0r1rwrrrrrMrr)4__doc__ __future__rrRrtorch.nn.functionalnn functionalr8rrrrrrasterr r rrrrrrrrHrVr[rrr~rrrrrrrrrr VIEWSr#r)r,rDrErFrGrOrRrTrrrr rrs#""""" FFFFFFFFFFFF********C" 6  XXXX (u ( ( ( ( ( (