docker 실행 후 random 시간이 흐르고 나면 "Failed to initialize NVML: Unknown Error"가 나는 현상

 Possible fix list

https://github.com/NVIDIA/nvidia-docker/issues/1671

https://gist.github.com/gengwg/55b3eb2bc22bcbd484fccbc0978484fc

https://bbs.archlinux.org/viewtopic.php?id=266915

https://github.com/NVIDIA/nvidia-docker/issues/1447




1) 문제 재구현: 아래 solution으로 해결 가능한 문제인지 진단

    - docker container 재시작 후 nvidia-smi 정상 동작 확인

    - 호스트 측 터미널에 "systemctl daemon-reload" 입력후 container 안에서 nvidia-smi error 발생 확인

2) 해결 방법: docker run 인자 수정

    - /dev/ 하위에 nvidia가 붙은 모든 instance들을 인자에 포함시켜 전달

    - 예시: docker run -it --name detr --gpus all -v /home/work/Desktop/:/data --shm-size 128G -p 11022:22 -p 11006:6006 --device /dev/nvidia0:/dev/nvidia0 --device /dev/nvidia1:/dev/nvidia1 --device /dev/nvidia2:/dev/nvidia2 --device /dev/nvidia3:/dev/nvidia3 --device /dev/nvidia4:/dev/nvidia4 --device /dev/nvidia5:/dev/nvidia5 --device /dev/nvidia6:/dev/nvidia6 --device /dev/nvidia7:/dev/nvidia7 --device /dev/nvidia-caps/ --device /dev/nvidiactl --device /dev/nvidia-modeset --device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools da_ddetr:latest

댓글

댓글 쓰기

이 블로그의 인기 게시물

sklearn tsne + matplotlib scatter

import os import torch import clip import matplotlib . pyplot as plt from PIL import Image from sklearn . manifold import TSNE from tqdm import tqdm from collections import defaultdict from random import shuffle   X = torch . cat ( X , dim = 0 ) X_emb = TSNE ( init = "pca" , perplexity = 30.0 ). fit_transform ( X . cpu ()) labels = [ "source" , "z_star/foggy-1" , "z_star/foggy-2" , "z_star/foggy-3" , "z_bar/foggy-1" , "z_bar/foggy-2" , "z_bar/foggy-3" ] fig , ax = plt . subplots ( 1 ) group_len = len ( zs ) for i in range ( 7 ): ax . scatter ( X_emb [ i * group_len :( i + 1 )* group_len , 0 ], X_emb [ i * group_len :( i + 1 )* group_len , 1 ], label = labels [ i ], s = 4 ) ax . legend () fig .savefig( "temp.png" )
자세한 내용 보기

Implementation of Focal Loss using Pytorch

import torch import torch.nn as nn class BinaryFocalLossWithLogits(nn.Module): def __init__ ( self , gamma= 0 , alpha_dominant= None, reduce= True ): super (BinaryFocalLossWithLogits , self ). __init__ () self .gamma = gamma self .alpha = alpha_dominant self .reduce = reduce self .BCELoss = nn.BCEWithLogitsLoss( reduction = 'mean' if self .reduce else 'none' ) def forward ( self , inputs , labels): loss = self .BCELoss(inputs , labels) pt = torch.exp(- 1 * loss) at = torch.tensor( list ( map ( lambda x: self .alpha if x == 1 else 1 - self .alpha , labels.view(- 1 ).tolist()))) loss = loss * (( 1 - pt) ** self .gamma) * at if self .reduce: return torch.mean(loss) else : return loss usage: criterion = BinaryFocalLossWithLogits( gamma = 2 , alpha_dominant = 0.1 , reduce = False ) labels = torch.tensor([[ 1. , 0. , 0. ]]) output = torch.tensor([[- 0...
자세한 내용 보기