解决RuntimeError: CUDA error: an illegal memory access was encountered问题

用pytorch在多卡训练transformers的时候出现了以下问题：

RuntimeError: CUDA error: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f7b645511e2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7f7b6479ff92 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f7b6453f9cd in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x540ae2 (0x7f7bb00d7ae2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x540b86 (0x7f7bb00d7b86 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: /usr/bin/python() [0x554316]
frame #6: /usr/bin/python() [0x585d2c]
frame #7: /usr/bin/python() [0x585d2c]
frame #8: /usr/bin/python() [0x573087]
frame #9: /usr/bin/python() [0x5542f2]
frame #10: /usr/bin/python() [0x586108]
frame #11: /usr/bin/python() [0x5aa838]
frame #12: /usr/bin/python() [0x5aa84e]
frame #13: /usr/bin/python() [0x5aa84e]
frame #14: /usr/bin/python() [0x56e336]
frame #15: PyDict_SetItemString + 0x156 (0x575256 in /usr/bin/python)
frame #16: PyImport_Cleanup + 0x76 (0x4ed566 in /usr/bin/python)
frame #17: Py_FinalizeEx + 0x5e (0x637efe in /usr/bin/python)
frame #18: Py_Main + 0x37e (0x638f5e in /usr/bin/python)
frame #19: main + 0xe0 (0x4a6f10 in /usr/bin/python)
frame #20: __libc_start_main + 0xe7 (0x7f7bb3412b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #21: _start + 0x2a (0x5afa0a in /usr/bin/python)