Subject: StandAlone job on k8s fails with "Unknown method truncate" on restore


That log does not appear. It looks like we have egg and chicken issue.

2019-02-15 16:49:15,045 DEBUG org.apache.hadoop.hdfs.DFSClient
                - Connecting to datanode 10.246.221.10:50010

2019-02-15 16:49:15,045 DEBUG
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  -
SASL client skipping handshake in unsecured configuration for

addr = /10.246.221.10, datanodeId = DatanodeInfoWithStorage[
10.246.221.10:50010,DS-c57a7667-f697-4f03-9fb1-532c5b82a9e8,DISK]

2019-02-15 16:49:15,072 DEBUG
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory              -
Instantiating for file system scheme hdfs Hadoop File System
org.apache.hadoop.hdfs.DistributedFileSystem

2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
                - dfs.client.use.legacy.blockreader.local = false

2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
                - dfs.client.read.shortcircuit = false

2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
                - dfs.client.domain.socket.data.traffic = false

2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal
                - dfs.domain.socket.path =

2019-02-15 16:49:15,076 DEBUG org.apache.hadoop.io.retry.RetryUtils
                - multipleLinearRandomRetry = null

2019-02-15 16:49:15,076 DEBUG org.apache.hadoop.ipc.Client
                - getting client out of cache:
org.apache.hadoop.ipc.Client@31920ade

2019-02-15 16:49:15,076 DEBUG
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil  -
DataTransferProtocol not using SaslPropertiesResolver, no QOP found in
configuration for dfs.data.transfer.protection

2019-02-15 16:49:15,080 INFO
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets  - Subtask
3 initializing its state (max part counter=58).

2019-02-15 16:49:15,081 DEBUG
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets  - Subtask
3 restoring: BucketState for
bucketId=ls_kraken_events/dt=2019-02-14/evt=ad_fill and
bucketPath=hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill,
has open part file created @ 1550247946437

2019-02-15 16:49:15,085 DEBUG org.apache.hadoop.ipc.Client
                - IPC Client (1270836494) connection to
nn-crunchy.bf2.tumblr.net/10.246.199.154:8020 from root sending #56

2019-02-15 16:49:15,188 DEBUG org.apache.hadoop.ipc.Client
                - IPC Client (1270836494) connection to
nn-crunchy.bf2.tumblr.net/10.246.199.154:8020 from root got value #56

2019-02-15 16:49:15,196 INFO  org.apache.flink.runtime.taskmanager.Task
                - Source: Custom Source -> (Sink: Unnamed, Process ->
Timestamps/Watermarks) (4/4) (f73403ac4763c99e6a244cba3797f7e9) switched
from RUNNING to FAILED.

java.io.IOException: Missing data in tmp file:
hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill/
.part-3-32.inprogress.da2a75d1-0c83-47bc-9c83-950360c55c86

        at
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.<init>(HadoopRecoverableFsDataOutputStream.java:93)
I do see
2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
      -  Current Hadoop/Kerberos user: root

2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
      -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.181-b13

2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
      -  Maximum heap size: 1204 MiBytes

2019-02-15 16:47:33,582 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
      -  JAVA_HOME: /docker-java-home

2019-02-15 16:47:33,585 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
      -  Hadoop version: 2.7.5

which has to be expected given that we are running the hadoop27flink 1.7.1
version.

Does it make sense to go with a hadoop less version and inject the required
jar files ?  Has that been done by anyone ?
On Fri, Feb 15, 2019 at 2:33 AM Yun Tang <[EMAIL PROTECTED]> wrote: