Use Paimon Hive Catalog with hiveconf configured will throw No FileSystem for scheme: file

[luoyuxia]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

If configure the following in server.yaml

datalake.format: paimon

datalake.paimon.metastore: hive
datalake.paimon.url: thrift://xxxx:9093
datalake.paimon.warehouse: hdfs://xxx/xxx
datalake.paimon.hadoop-conf-dir: file:///opt/hive-conf
datalake.paimon.hive-conf-dir: file:///opt/hive-conf

Then, start the coordinator, the following exception will be thrown:

Caused by: java.io.IOException: No FileSystem for scheme: file
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) ~[?:?]
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849) ~[?:?]
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831) ~[?:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) ~[?:?]
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356) ~[?:?]
        at org.apache.paimon.hive.HiveCatalog.createHiveConf(HiveCatalog.java:1354) ~[?:?]
        at org.apache.paimon.hive.HiveCatalog.createHiveConf(HiveCatalog.java:1412) ~[?:?]
        at org.apache.paimon.hive.HiveCatalog.createHiveCatalog(HiveCatalog.java:1379) ~[?:?]
        at org.apache.paimon.hive.HiveCatalogFactory.create(HiveCatalogFactory.java:37) ~[?:?]
        at org.apache.paimon.catalog.CatalogFactory.createUnwrappedCatalog(CatalogFactory.java:84) ~[?:?]
        at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:71) ~[?:?]
        at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:67) ~[?:?]

We have put flink-shaded-hadoop-2-uber-2.8.3-10.0.jar in /plugins/paimon/ , which should include LocalFileSystem to support file scheme.

The reason why the exception happens is that:
org.apache.hadoop.fs.Path.getFileSystem will use thread current classloader(which is app classloader) to load filesystem, but obviously it's not in app classloader. If I also copy flink-shaded-hadoop-2-uber-2.8.3-10.0.jar into FLUSS_HOME/lib.
It will throw class not cast exception since hadoop classes in both /plugins/paimon/ and FLUSS_HOME/lib.

Solution

work around

Now, I found a workround:

• Move /plugins/paimon/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar to FLUSS_HOME/lib
• Add a option in server.yaml:

plugin.classloader.parent-first-patterns.default: java.,com.alibaba.fluss.,javax.annotation.,org.slf4j,org.apache.log4j,org.apache.logging,org.apache.commons.logging,ch.qos.logback,org.apache.hadoop.

To solve this, while calling pluginable lake catalog, we may need to set the current thread classloader to the classloader of lake catalog(which is pluginClassloader including hadoop classes).

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[beryllw]

Is this resolved? Could we also support using the machine's Hadoop environment?

[luoyuxia]

@beryllw Thanks for take it. I will create another issue for using the machine's Hadoop environment. In this issue, we can just use plugin classloader as thread current classloader. But we can make it opt-in to enable delegate to app classloader for hadoop classes.

[wuchong]

What's the final solution of this issue? It seems very strange to me that we put a flink hadoop dependency into default fluss classpath.

[luoyuxia]

What's the final solution of this issue? It seems very strange to me that we put a flink hadoop dependency into default fluss classpath.

Wrap the LakeStoragePlugin to set the Thread current classloader to the plugin classloader for all operations of LakeStoragePlugin just like we do in PluginFileSystemWrapper. So that it can find the LocalFileSystem since the thread current classloader to find the class is PluginClassloader, which include the class now.