jE'RdZddlZddlZddlZddlZddlZdZdZdZdZ dZ dxa a d a d Zd \ZZZd ZejejejeZd ZdZdZdZddZdZdZdZdZ dZ!dZ"dZ#dZ$ddZ%dS)aCUDA memory settings applied when WanGP starts, before anything initializes CUDA: the VRAM allocator (Configuration > RAM/VRAM Management or --vram-allocator), its debug mode (--vram-debug) and the per-thread stack reserve of the CUDA context; the RAM debug mode (--ram-debug) and the MMGP RAM allocator (Configuration > RAM/VRAM Management or --ram-allocator); and the folder of the driver's compute cache (Configuration > Performance).Nvram_allocator)defaultvmm vmm_spill ram_allocator)rmmgprFr) compute_cachecompute_cache_path ComputeCachelc~tjt|pt S)zHThe folder of the compute cache: a relative path is in the WanGP folder.)ospathjoin WANGP_ROOTstripCOMPUTE_CACHE_PATH_DEFAULT)rs :C:\Users\johnn\OneDrive\Documents\AI\shared\cuda_memory.pycompute_cache_folderrs& 7<< DJJLL$N4N O OOctdvS)zThe MMGP VRAM allocator runs: without the fragmentation the VAE tiling presets were made against, VAEs offer two levels, Default (fast, about 2-3 GB of VRAM) and Low VRAM (Configuration > RAM/VRAM Management > VAE Tiling, values 0 and 3).rr)VRAM_ALLOCATORrrmmgp_vram_allocatorr!s 1 11rcddl}|jjduo;|jo"|jddkS)a NVIDIA Blackwell GPUs (RTX 50xx, B200...): libraries ship their code in a form that the driver optimizes for the exact GPU when it loads it, about 15 s per launch, and caches; older GPUs run the shipped code as is. ROCm reports AMD architectures as capabilities.rN )torchversionhipcuda is_availableget_device_capability)rs rcompute_cache_usefulr$'sOLLL =  $ r)@)@)B)B ruzGgGgGiGijkGlprGrrrct|D]d\}}||kr#|dzt|kr ||dzcS||dzr|dddcSedS)N=) enumeratelen startswithsplit)argvoptionindexargs r _argv_valuer0.soo(( s &==UQYT22 ? " " " >>&3, ' ' (99S!$$Q' ' ' ' ( 4rct|d}|r>tjtj||gng|gzD]t}tj|rSt |d5}tj| ||cdddcS#1swxYwYu|S)Nz--configzutf-8)encoding) r0rrrabspathisfileopenjsonloadget)r,config_filenamekeyr config_dirrreaders r _config_valuer=7sT:..JQ[c"',,rwz::OLLMMachwgxx;; 7>>$   ;dW--- ;y((,,S':: ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; Ns (CC C cLt||}|t|||n|SN)r0r=)r,r9r-r:values r _requestedrAAs, f % %E8= = 4 4 45Prct||tdsdStt||tt} t j|dn.#t$r!}td|d|dYd}~dSd}~wwxYw|ttct j d<t j d<td |dS) NFT)exist_okz[CUDA] Compute Cache folder z not available (z,): the NVIDIA driver's default cache is usedCUDA_CACHE_PATHCUDA_CACHE_MAXSIZEz[CUDA] Compute Cache: ) r=COMPUTE_CACHE_KEYrCOMPUTE_CACHE_PATH_KEYrrmakedirsOSErrorprintstrCOMPUTE_CACHE_MAX_BYTESenviron)r,r9rerrors r_apply_compute_cacherOFs 0A5 I I  dOE[]w x x y yD D4(((((  vTvv5vvvwwwGKCPgLhLhCBJ !2:.B#C )4 ) )*****sA B%BBc4t||dtpdS)Nz--vram-allocatorr)rAVRAM_ALLOCATOR_KEY)r,r9s rrequested_vram_allocatorrRVs dO-?AS T T cXccrcddl}t||dtpt}|tvrt d|dt|dkrUddlm} |tdn*#t$r}td|d Yd}~nd}~wwxYw|j j t|||j j *|jrt!|||t#|d }|r$|d st'|nd |vrd nd}|dkr4dd lm}||datd|dddSdS)Nrz--ram-allocatorzUnknown RAM allocator : expected one of rramzN[RAM] MMGP RAM Allocator: the RAM of freed CPU tensors goes back to the systemz([RAM] MMGP RAM Allocator not available (z"): PyTorch's CPU allocator is usedz --ram-debug- ram_debugmin_mbTz[RAM] Debug mode: a report of where the RAM goes is written after each generation and after each model release in /ram_debug (allocations of gz$ MB and more traced to their source))rrARAM_ALLOCATOR_KEYRAM_ALLOCATOR_DEFAULTRAM_ALLOCATOR_CHOICES ValueErrormmgp.allocatorrVinstallrJ RuntimeErrorrr rOr!r"_apply_cuda_settingsr0r*floatrZstart _ram_debug) r,r9rrrVrN debug_valuedebug_mbrZs rapply_startup_settingsrkZsLLLt_6GIZ[[t_tM111l-llUjllmmm&&&&&& d KKMMM b c c c c h h h fUfff g g g g g g g g h } T?333 } UZ%<%<%>%> T?E:::dM22K%0|9O9OPS9T9T|u[!!!`mqu`u`uZ\Z\{|H!||,,,,,,x(((  R!)QRRR S S S S S |sA77 BBBct||}|tvrtd|dt|dvrhddlm} ||dkt d|dkrd nd zn,#t$r}t d |d d }Yd}~nd}~wwxYw|att|dpd}|dkrE|dvrtdddl m }| |dat d|ddtt jdt&}|dkr|ddt*jdkrt/jdnt/jd} | dt/j|} | dkrtd|d| dSdS)NzUnknown VRAM allocator rTrr) allocatorr)spillz$[VRAM] MMGP Optimized VRAM Allocatorz%, spilling into RAM when VRAM is fullz4[VRAM] MMGP Optimized VRAM Allocator not available (z): PyTorch's allocator is usedrz --vram-debugzH--vram-debug needs the MMGP allocator: --vram-allocator vmm or vmm_spilldebugr[Tz"[VRAM] Debug mode: allocations of r]zX MB and more recorded, a report is written after each generation in /vram_debugWANGP_CUDA_STACK_BYTESr&r!)devicewin32z nvcuda.dllz libcuda.so.1z#cuCtxSetLimit(CU_LIMIT_STACK_SIZE, z) failed with CUDA error )rRVRAM_ALLOCATOR_CHOICESrarrmrcrJrdrrfr0rbrqrg _vram_debugintrrMr8CUDA_STACK_BYTESzerossysplatformctypesWinDLLCDLL cuCtxSetLimitc_size_t) r,r9rrmrrNrjrq stack_bytesdriverresults rrerevsl(??I...j9jjRhjjkkk(((444444 J  " "k)A " B B B 8gptggBBc^tsdSddlm}tj|dt jdd|pdd}||| td |d tj |dd dS) z{With --vram-debug: the report of the generation that just ended, in output_dir/vram_debug; then new peaks for the next one.Nrrp vram_debug%Y-%m-%d-%Hh%Mm%Ss_ generation.jsonz[VRAM debug] Report: (summary: z.md)) rvrbrqrrrtimestrftimereportresetrJsplitext) output_dirlabelrqrs rwrite_vram_debug_reportrs $$$$$$ 7<< LT]CW5X5X2w2w[`[pdp2w2w2w x xD LL KKMMM R$ R R273C3CD3I3I!3L R R RSSSSSrc:ddlm}|dS)zThe end of a queue: the freed blocks the MMGP RAM allocator kept for reuse go back to the system (between generations of the same model they are kept: the next one reuses them).rrUN)rbrVreleaserUs rrelease_ram_cachers'#"""""KKMMMMMrcNtrddlm}||dSdS)zIWith --ram-debug: the RAM that follows counts for the peak of phase name.rrYN)rhrbrZmark)namerZs rram_debug_markrs>,,,,,,trafter generationcdtsdSddlm}tj|dt jdd|pdd}|||| | td |d tj |dd dS) zWith --ram-debug: a census of the RAM now (census_label) and the peaks of the phases since the previous report, in output_dir/ram_debug.NrrYrZrrrr)rsnapshot_labelz[RAM debug] Report: rzJ.md); the latest report includes the RAM after each report since the start) rhrbrZrrrrrrrrJr)rr census_labelrZrs rwrite_ram_debug_reportrs (((((( 7<< KDMBV4W4W1v1vZ_Zoco1v1v1v w wD T|DDD OO X X X"'2B2B42H2H2K X X XYYYYYrr?)r)&__doc__r|r6rrzrrQrur^r`r_rvrhrrxrFrGrrLrdirnamer3__file__rrrr$r0r=rArOrRrkrerrrrrrrrs00 %8#+ jH}E)+E! W__RW__RW__X-F-FGG H H PPP 222 sssQQQ + + + dddSSS8uuuBTTTYYYYYYr