问题详情:
/arrow/cpp/src/arrow/filesystem/:51: Failed to disconnect hdfs client: IOError: HDFS hdfsFS::Disconnect failed, errno: 9 (Bad file descriptor)
Traceback (most recent call last):
File "/home/tdops/fucheng.pan/ray-code/read.py", line 15, in
print(fs.FileSystem.from_uri("hdfs:"))
File "pyarrow/_fs.pyx", line 347, in pyarrow._fs.FileSystem.from_uri
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: HDFS connection failed
dask及相关库的版本如下:
pyarrow 15.0.2
dask 2024.4.0
dask-expr 1.0.6
fsspec 2024.3.1
解决办法:
先将 fsspec 降至 2022.8.2 版本,如果再不行,将pyarrow降至 7.0.0 解决