调参侠看过来！两个提高深度学习训练效率的绝技
|
horovod安装 建议安装docker的horovod，省去安装环境的麻烦。horovod依赖NCCL 2 open MPI $ mkdir horovod-docker-gpu $ wget -O horovod-docker-gpu/Dockerfile https://raw.githubusercontent.com/horovod/horovod/master/Dockerfile.gpu $ docker build -t horovod:latest horovod-docker-gpu 机器worker机器之间ssh打通 修改训练代码 horovod支持tf,keras,pytorch和mxnet等不同的深度学习框架。以keras为例，修改主要6个步骤 （1） 初始化：hvd.init() （2）分配GPU计算资源：config.gpu_options.visible_device_list = str(hvd.local_rank())（3）分布式的优化器来实现参数的分布式更新：opt = hvd.DistributedOptimizer(opt)（4）定义所有worker模型初始化一致性 hvd.callbacks.BroadcastGlobalVariablesCallback(0)（5）模型保存在某一个worker from __future__ import print_function import keras from keras.datasets import mnist from keras.models import Sequential from keras.layers import Dense, Dropout, Flatten from keras.layers import Conv2D, MaxPooling2D from keras import backend as K import math import tensorflow as tf import horovod.keras as hvd # Horovod: initialize Horovod. hvd.init() # Horovod: pin GPU to be used to process local rank (one GPU per process) config = tf.ConfigProto() config.gpu_options.allow_growth = True config.gpu_options.visible_device_list = str(hvd.local_rank()) K.set_session(tf.Session(configconfig=config)) batch_size = 128 num_classes = 10 # Horovod: adjust number of epochs based on number of GPUs. epochs = int(math.ceil(12.0 / hvd.size())) # Input image dimensions img_rows, img_cols = 28, 28 # The data, shuffled and split between train and test sets (x_train, y_train), (x_test, y_test) = mnist.load_data() if K.image_data_format() == 'channels_first': x_trainx_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols) x_testx_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols) input_shape = (1, img_rows, img_cols) else: x_trainx_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1) x_testx_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1) input_shape = (img_rows, img_cols, 1) x_trainx_train = x_train.astype('float32') x_testx_test = x_test.astype('float32') x_train /= 255 x_test /= 255 print('x_train shape:', x_train.shape) print(x_train.shape[0], 'train samples') print(x_test.shape[0], 'test samples') # Convert class vectors to binary class matrices y_train = keras.utils.to_categorical(y_train, num_classes) y_test = keras.utils.to_categorical(y_test, num_classes) model = Sequential() model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shapeinput_shape=input_shape)) model.add(Conv2D(64, (3, 3), activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(128, activation='relu')) model.add(Dropout(0.5)) model.add(Dense(num_classes, activation='softmax')) # Horovod: adjust learning rate based on number of GPUs. opt = keras.optimizers.Adadelta(1.0 * hvd.size()) # Horovod: add Horovod Distributed Optimizer. opt = hvd.DistributedOptimizer(opt) model.compile(loss=keras.losses.categorical_crossentropy, optoptimizer=opt, metrics=['accuracy']) callbacks = [ # Horovod: broadcast initial variable states from rank 0 to all other processes. # This is necessary to ensure consistent initialization of all workers when # training is started with random weights or restored from a checkpoint. hvd.callbacks.BroadcastGlobalVariablesCallback(0), ] # Horovod: save checkpoints only on worker 0 to prevent other workers from corrupting them. if hvd.rank() == 0: callbacks.append(keras.callbacks.ModelCheckpoint('./checkpoint-{epoch}.h5')) model.fit(x_train, y_train, batch_sizebatch_size=batch_size, callbackscallbacks=callbacks, epochsepochs=epochs, verbose=1, validation_data=(x_test, y_test)) score = model.evaluate(x_test, y_test, verbose=0) print('Test loss:', score[0]) print('Test accuracy:', score[1]) 利用horovodrun 执行分布式训练 horovodrun -np 16 -H server1:4,server2:4,server3:4,server4:4 python train.py 5. 总结 本文分享了通过GPU利用率和分布式训练Horovod框架来提升深度学习训练。 并行CPU加载数据和预处理，让GPU不再等待CPU 采用Horovod让数据并行来提高大数据量的训练的迭代时间
（编辑：惠州站长网） 【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容! |
