Batch size: incrementally increase your batch size until you go out of memory.The garbage collector won't release them until they go out of scope. Tensors usage: minimise the number of tensors that you create.Usually, you fix a given number of decoding steps that is reasonable for your dataset. RNN decoder maximum steps: if you're using an RNN decoder in your architecture, avoid looping for a big number of steps.Linear layers that transform a big input tensor (e.g., size 1000) in another big output tensor (e.g., size 1000) will require a matrix whose size is (1000, 1000). Modules parameters: check the number of dimensions for your modules.It might be for a number of reasons that I try to report in the following list: So, in that case, you can explicitly delete variables after performing optimizer.step() for one_epoch in range(100):ĭel intermediate_variable1,intermediate_variable2. If you use for loop in training code,ĭata can be sustained until entire for loop ends. In that case, you need to use float() like following siteĮven if docs guides with float(), in case of me, item() also worked like entire_loss=0.0ģ. It can accumulate gradient continuously in your gradient graph. In that situation, your code can be located under with torch.no_grad(): You don't need to calculate gradients for forward and backward phase. When you only perform validation not training,
0 kommentar(er)
