William Shen (JIRA)
2018-12-06 20:07:00 UTC
William Shen created HBASE-21563:
------------------------------------
Summary: HBase Get Encounters java.lang.IndexOutOfBoundsException
Key: HBASE-21563
URL: https://issues.apache.org/jira/browse/HBASE-21563
Project: HBase
Issue Type: Bug
Components: HFile
Affects Versions: 1.2.0
Reporter: William Shen
Attachments: 67a04bc049be4f58afecdcc0a3ba62ca.tar.gz
We've recently encountered issue retrieving data from our HBase cluster, and have not had much luck troubleshooting the issue. We narrowed down our issue to a single GET, which appears to be caused by FastDiffDeltaEncoder.java running into java.lang.IndexOutOfBoundsException.
Perhaps there is a bug on a corner case for FastDiffDeltaEncoder?
We are running 1.2.0-cdh5.9.2, and the GET in question is:
{noformat}
hbase(main):004:0> get 'qa2.ADGROUPS', "\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"
COLUMN CELL
ERROR: java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkBounds(Buffer.java:567)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
... 3 more {noformat}
Likewise, running {{ hbase hfile -f -p }} on the specific hfile, a subset of kv pairs were printed until the program hits the following exception and crashes:
{noformat}
Exception in thread "main" java.lang.RuntimeException: Unknown code 65
at org.apache.hadoop.hbase.KeyValue$Type.codeToType(KeyValue.java:259)
at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:1246)
at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$ClonedSeekerState.toString(BufferedDataBlockEncoder.java:506)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:382)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:316)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:677)
{noformat}
I have attached the HFile related to this issue for debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
------------------------------------
Summary: HBase Get Encounters java.lang.IndexOutOfBoundsException
Key: HBASE-21563
URL: https://issues.apache.org/jira/browse/HBASE-21563
Project: HBase
Issue Type: Bug
Components: HFile
Affects Versions: 1.2.0
Reporter: William Shen
Attachments: 67a04bc049be4f58afecdcc0a3ba62ca.tar.gz
We've recently encountered issue retrieving data from our HBase cluster, and have not had much luck troubleshooting the issue. We narrowed down our issue to a single GET, which appears to be caused by FastDiffDeltaEncoder.java running into java.lang.IndexOutOfBoundsException.
Perhaps there is a bug on a corner case for FastDiffDeltaEncoder?
We are running 1.2.0-cdh5.9.2, and the GET in question is:
{noformat}
hbase(main):004:0> get 'qa2.ADGROUPS', "\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"
COLUMN CELL
ERROR: java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkBounds(Buffer.java:567)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
... 3 more {noformat}
Likewise, running {{ hbase hfile -f -p }} on the specific hfile, a subset of kv pairs were printed until the program hits the following exception and crashes:
{noformat}
Exception in thread "main" java.lang.RuntimeException: Unknown code 65
at org.apache.hadoop.hbase.KeyValue$Type.codeToType(KeyValue.java:259)
at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:1246)
at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$ClonedSeekerState.toString(BufferedDataBlockEncoder.java:506)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:382)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:316)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:255)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:677)
{noformat}
I have attached the HFile related to this issue for debugging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)