Bing Li
2013-02-04, 20:20
Harsh J
2013-02-04, 20:21
Ted Yu
2013-02-04, 20:25
Bing Li
2013-02-04, 20:32
Haijia Zhou
2013-02-04, 20:42
Adrien Mogenet
2013-02-04, 21:13
Nicolas Liochon
2013-02-04, 21:31
Bing Li
2013-02-04, 22:40
Nicolas Liochon
2013-02-04, 22:49
Bing Li
2013-02-05, 16:54
lars hofhansl
2013-02-06, 05:05
Bing Li
2013-02-06, 06:36
lars hofhansl
2013-02-06, 07:44
Adrien Mogenet
2013-02-06, 07:45
Bing Li
2013-02-06, 10:31
lars hofhansl
2013-02-06, 18:54
Bing Li
2013-02-07, 08:10
lars hofhansl
2013-02-07, 17:24
Dear all,
When writing data into HBase, sometimes I got exceptions. I guess they
might be caused by concurrent writings. But I am not sure.
My question is whether it is necessary to put "synchronized" before
the writing methods? The following lines are the sample code.
I think the directive, synchronized, must lower the performance of
writing. Sometimes concurrent writing is needed in my system.
Thanks so much!
Best wishes,
Bing
public synchronized void AddDomainNodeRanks(String domainKey, int
timingScale, Map<String, Double> nodeRankMap)
{
List<Put> puts = new ArrayList<Put>();
Put domainKeyPut;
Put timingScalePut;
Put nodeKeyPut;
Put rankPut;
byte[] domainNodeRankRowKey;
for (Map.Entry<String, Double> nodeRankEntry : nodeRankMap.entrySet())
{
domainNodeRankRowKey Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
domainKeyPut = new Put(domainNodeRankRowKey);
domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
Bytes.toBytes(domainKey));
puts.add(domainKeyPut);
timingScalePut = new Put(domainNodeRankRowKey);
timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
Bytes.toBytes(timingScale));
puts.add(timingScalePut);
nodeKeyPut = new Put(domainNodeRankRowKey);
nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
Bytes.toBytes(nodeRankEntry.getKey()));
puts.add(nodeKeyPut);
rankPut = new Put(domainNodeRankRowKey);
rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
Bytes.toBytes(nodeRankEntry.getValue()));
puts.add(rankPut);
}
try
{
this.rankTable.put(puts);
}
catch (IOException e)
{
e.printStackTrace();
}
}
When writing data into HBase, sometimes I got exceptions. I guess they
might be caused by concurrent writings. But I am not sure.
My question is whether it is necessary to put "synchronized" before
the writing methods? The following lines are the sample code.
I think the directive, synchronized, must lower the performance of
writing. Sometimes concurrent writing is needed in my system.
Thanks so much!
Best wishes,
Bing
public synchronized void AddDomainNodeRanks(String domainKey, int
timingScale, Map<String, Double> nodeRankMap)
{
List<Put> puts = new ArrayList<Put>();
Put domainKeyPut;
Put timingScalePut;
Put nodeKeyPut;
Put rankPut;
byte[] domainNodeRankRowKey;
for (Map.Entry<String, Double> nodeRankEntry : nodeRankMap.entrySet())
{
domainNodeRankRowKey Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
domainKeyPut = new Put(domainNodeRankRowKey);
domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
Bytes.toBytes(domainKey));
puts.add(domainKeyPut);
timingScalePut = new Put(domainNodeRankRowKey);
timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
Bytes.toBytes(timingScale));
puts.add(timingScalePut);
nodeKeyPut = new Put(domainNodeRankRowKey);
nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
Bytes.toBytes(nodeRankEntry.getKey()));
puts.add(nodeKeyPut);
rankPut = new Put(domainNodeRankRowKey);
rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
Bytes.toBytes(nodeRankEntry.getValue()));
puts.add(rankPut);
}
try
{
this.rankTable.put(puts);
}
catch (IOException e)
{
e.printStackTrace();
}
}
What exceptions do you actually receive - can you send them here?
Knowing that is key to addressing your issue.
On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> When writing data into HBase, sometimes I got exceptions. I guess they
> might be caused by concurrent writings. But I am not sure.
>
> My question is whether it is necessary to put "synchronized" before
> the writing methods? The following lines are the sample code.
>
> I think the directive, synchronized, must lower the performance of
> writing. Sometimes concurrent writing is needed in my system.
>
> Thanks so much!
>
> Best wishes,
> Bing
>
> public synchronized void AddDomainNodeRanks(String domainKey, int
> timingScale, Map<String, Double> nodeRankMap)
> {
> List<Put> puts = new ArrayList<Put>();
> Put domainKeyPut;
> Put timingScalePut;
> Put nodeKeyPut;
> Put rankPut;
>
> byte[] domainNodeRankRowKey;
>
> for (Map.Entry<String, Double> nodeRankEntry : nodeRankMap.entrySet())
> {
> domainNodeRankRowKey > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
>
> domainKeyPut = new Put(domainNodeRankRowKey);
> domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> Bytes.toBytes(domainKey));
> puts.add(domainKeyPut);
>
> timingScalePut = new Put(domainNodeRankRowKey);
> timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> Bytes.toBytes(timingScale));
> puts.add(timingScalePut);
>
> nodeKeyPut = new Put(domainNodeRankRowKey);
> nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> Bytes.toBytes(nodeRankEntry.getKey()));
> puts.add(nodeKeyPut);
>
> rankPut = new Put(domainNodeRankRowKey);
> rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> Bytes.toBytes(nodeRankEntry.getValue()));
> puts.add(rankPut);
> }
>
> try
> {
> this.rankTable.put(puts);
> }
> catch (IOException e)
> {
> e.printStackTrace();
> }
> }
--
Harsh J
Knowing that is key to addressing your issue.
On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear all,
>
> When writing data into HBase, sometimes I got exceptions. I guess they
> might be caused by concurrent writings. But I am not sure.
>
> My question is whether it is necessary to put "synchronized" before
> the writing methods? The following lines are the sample code.
>
> I think the directive, synchronized, must lower the performance of
> writing. Sometimes concurrent writing is needed in my system.
>
> Thanks so much!
>
> Best wishes,
> Bing
>
> public synchronized void AddDomainNodeRanks(String domainKey, int
> timingScale, Map<String, Double> nodeRankMap)
> {
> List<Put> puts = new ArrayList<Put>();
> Put domainKeyPut;
> Put timingScalePut;
> Put nodeKeyPut;
> Put rankPut;
>
> byte[] domainNodeRankRowKey;
>
> for (Map.Entry<String, Double> nodeRankEntry : nodeRankMap.entrySet())
> {
> domainNodeRankRowKey > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
>
> domainKeyPut = new Put(domainNodeRankRowKey);
> domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> Bytes.toBytes(domainKey));
> puts.add(domainKeyPut);
>
> timingScalePut = new Put(domainNodeRankRowKey);
> timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> Bytes.toBytes(timingScale));
> puts.add(timingScalePut);
>
> nodeKeyPut = new Put(domainNodeRankRowKey);
> nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> Bytes.toBytes(nodeRankEntry.getKey()));
> puts.add(nodeKeyPut);
>
> rankPut = new Put(domainNodeRankRowKey);
> rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> Bytes.toBytes(nodeRankEntry.getValue()));
> puts.add(rankPut);
> }
>
> try
> {
> this.rankTable.put(puts);
> }
> catch (IOException e)
> {
> e.printStackTrace();
> }
> }
--
Harsh J
Bing:
Use pastebin.com instead of attaching exception report.
What version of HBase are you using ?
Thanks
On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> What exceptions do you actually receive - can you send them here?
> Knowing that is key to addressing your issue.
>
> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> > Dear all,
> >
> > When writing data into HBase, sometimes I got exceptions. I guess they
> > might be caused by concurrent writings. But I am not sure.
> >
> > My question is whether it is necessary to put "synchronized" before
> > the writing methods? The following lines are the sample code.
> >
> > I think the directive, synchronized, must lower the performance of
> > writing. Sometimes concurrent writing is needed in my system.
> >
> > Thanks so much!
> >
> > Best wishes,
> > Bing
> >
> > public synchronized void AddDomainNodeRanks(String domainKey, int
> > timingScale, Map<String, Double> nodeRankMap)
> > {
> > List<Put> puts = new ArrayList<Put>();
> > Put domainKeyPut;
> > Put timingScalePut;
> > Put nodeKeyPut;
> > Put rankPut;
> >
> > byte[] domainNodeRankRowKey;
> >
> > for (Map.Entry<String, Double> nodeRankEntry :
> nodeRankMap.entrySet())
> > {
> > domainNodeRankRowKey > > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> >
> > domainKeyPut = new Put(domainNodeRankRowKey);
> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> > Bytes.toBytes(domainKey));
> > puts.add(domainKeyPut);
> >
> > timingScalePut = new Put(domainNodeRankRowKey);
> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> > Bytes.toBytes(timingScale));
> > puts.add(timingScalePut);
> >
> > nodeKeyPut = new Put(domainNodeRankRowKey);
> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> > Bytes.toBytes(nodeRankEntry.getKey()));
> > puts.add(nodeKeyPut);
> >
> > rankPut = new Put(domainNodeRankRowKey);
> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> > Bytes.toBytes(nodeRankEntry.getValue()));
> > puts.add(rankPut);
> > }
> >
> > try
> > {
> > this.rankTable.put(puts);
> > }
> > catch (IOException e)
> > {
> > e.printStackTrace();
> > }
> > }
>
>
>
> --
> Harsh J
>
Use pastebin.com instead of attaching exception report.
What version of HBase are you using ?
Thanks
On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> What exceptions do you actually receive - can you send them here?
> Knowing that is key to addressing your issue.
>
> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> > Dear all,
> >
> > When writing data into HBase, sometimes I got exceptions. I guess they
> > might be caused by concurrent writings. But I am not sure.
> >
> > My question is whether it is necessary to put "synchronized" before
> > the writing methods? The following lines are the sample code.
> >
> > I think the directive, synchronized, must lower the performance of
> > writing. Sometimes concurrent writing is needed in my system.
> >
> > Thanks so much!
> >
> > Best wishes,
> > Bing
> >
> > public synchronized void AddDomainNodeRanks(String domainKey, int
> > timingScale, Map<String, Double> nodeRankMap)
> > {
> > List<Put> puts = new ArrayList<Put>();
> > Put domainKeyPut;
> > Put timingScalePut;
> > Put nodeKeyPut;
> > Put rankPut;
> >
> > byte[] domainNodeRankRowKey;
> >
> > for (Map.Entry<String, Double> nodeRankEntry :
> nodeRankMap.entrySet())
> > {
> > domainNodeRankRowKey > > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> >
> > domainKeyPut = new Put(domainNodeRankRowKey);
> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> > Bytes.toBytes(domainKey));
> > puts.add(domainKeyPut);
> >
> > timingScalePut = new Put(domainNodeRankRowKey);
> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> > Bytes.toBytes(timingScale));
> > puts.add(timingScalePut);
> >
> > nodeKeyPut = new Put(domainNodeRankRowKey);
> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> > Bytes.toBytes(nodeRankEntry.getKey()));
> > puts.add(nodeKeyPut);
> >
> > rankPut = new Put(domainNodeRankRowKey);
> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> > Bytes.toBytes(nodeRankEntry.getValue()));
> > puts.add(rankPut);
> > }
> >
> > try
> > {
> > this.rankTable.put(puts);
> > }
> > catch (IOException e)
> > {
> > e.printStackTrace();
> > }
> > }
>
>
>
> --
> Harsh J
>
Dear Ted and Harsh,
I am sorry I didn't keep the exceptions. It occurred many days ago. My
current version is 0.92.
Now "synchronized" is removed. Is it correct?
I will test if such exceptions are raised. I will let you know.
Thanks!
Best wishes,
Bing
On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Bing:
> Use pastebin.com instead of attaching exception report.
>
> What version of HBase are you using ?
>
> Thanks
>
>
> On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> What exceptions do you actually receive - can you send them here?
>> Knowing that is key to addressing your issue.
>>
>> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
>> > Dear all,
>> >
>> > When writing data into HBase, sometimes I got exceptions. I guess they
>> > might be caused by concurrent writings. But I am not sure.
>> >
>> > My question is whether it is necessary to put "synchronized" before
>> > the writing methods? The following lines are the sample code.
>> >
>> > I think the directive, synchronized, must lower the performance of
>> > writing. Sometimes concurrent writing is needed in my system.
>> >
>> > Thanks so much!
>> >
>> > Best wishes,
>> > Bing
>> >
>> > public synchronized void AddDomainNodeRanks(String domainKey, int
>> > timingScale, Map<String, Double> nodeRankMap)
>> > {
>> > List<Put> puts = new ArrayList<Put>();
>> > Put domainKeyPut;
>> > Put timingScalePut;
>> > Put nodeKeyPut;
>> > Put rankPut;
>> >
>> > byte[] domainNodeRankRowKey;
>> >
>> > for (Map.Entry<String, Double> nodeRankEntry :
>> > nodeRankMap.entrySet())
>> > {
>> > domainNodeRankRowKey >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
>> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
>> >
>> > domainKeyPut = new Put(domainNodeRankRowKey);
>> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
>> > Bytes.toBytes(domainKey));
>> > puts.add(domainKeyPut);
>> >
>> > timingScalePut = new Put(domainNodeRankRowKey);
>> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
>> > Bytes.toBytes(timingScale));
>> > puts.add(timingScalePut);
>> >
>> > nodeKeyPut = new Put(domainNodeRankRowKey);
>> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
>> > Bytes.toBytes(nodeRankEntry.getKey()));
>> > puts.add(nodeKeyPut);
>> >
>> > rankPut = new Put(domainNodeRankRowKey);
>> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
>> > Bytes.toBytes(nodeRankEntry.getValue()));
>> > puts.add(rankPut);
>> > }
>> >
>> > try
>> > {
>> > this.rankTable.put(puts);
>> > }
>> > catch (IOException e)
>> > {
>> > e.printStackTrace();
>> > }
>> > }
>>
>>
>>
>> --
>> Harsh J
>
>
I am sorry I didn't keep the exceptions. It occurred many days ago. My
current version is 0.92.
Now "synchronized" is removed. Is it correct?
I will test if such exceptions are raised. I will let you know.
Thanks!
Best wishes,
Bing
On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Bing:
> Use pastebin.com instead of attaching exception report.
>
> What version of HBase are you using ?
>
> Thanks
>
>
> On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> What exceptions do you actually receive - can you send them here?
>> Knowing that is key to addressing your issue.
>>
>> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
>> > Dear all,
>> >
>> > When writing data into HBase, sometimes I got exceptions. I guess they
>> > might be caused by concurrent writings. But I am not sure.
>> >
>> > My question is whether it is necessary to put "synchronized" before
>> > the writing methods? The following lines are the sample code.
>> >
>> > I think the directive, synchronized, must lower the performance of
>> > writing. Sometimes concurrent writing is needed in my system.
>> >
>> > Thanks so much!
>> >
>> > Best wishes,
>> > Bing
>> >
>> > public synchronized void AddDomainNodeRanks(String domainKey, int
>> > timingScale, Map<String, Double> nodeRankMap)
>> > {
>> > List<Put> puts = new ArrayList<Put>();
>> > Put domainKeyPut;
>> > Put timingScalePut;
>> > Put nodeKeyPut;
>> > Put rankPut;
>> >
>> > byte[] domainNodeRankRowKey;
>> >
>> > for (Map.Entry<String, Double> nodeRankEntry :
>> > nodeRankMap.entrySet())
>> > {
>> > domainNodeRankRowKey >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
>> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
>> >
>> > domainKeyPut = new Put(domainNodeRankRowKey);
>> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
>> > Bytes.toBytes(domainKey));
>> > puts.add(domainKeyPut);
>> >
>> > timingScalePut = new Put(domainNodeRankRowKey);
>> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
>> > Bytes.toBytes(timingScale));
>> > puts.add(timingScalePut);
>> >
>> > nodeKeyPut = new Put(domainNodeRankRowKey);
>> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
>> > Bytes.toBytes(nodeRankEntry.getKey()));
>> > puts.add(nodeKeyPut);
>> >
>> > rankPut = new Put(domainNodeRankRowKey);
>> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
>> > Bytes.toBytes(nodeRankEntry.getValue()));
>> > puts.add(rankPut);
>> > }
>> >
>> > try
>> > {
>> > this.rankTable.put(puts);
>> > }
>> > catch (IOException e)
>> > {
>> > e.printStackTrace();
>> > }
>> > }
>>
>>
>>
>> --
>> Harsh J
>
>
Hi, Bing,
Not sure about your scenario but HTable class is not thread safe for
neither reads nor write.
If you consider writing/reading from a table in a multiple-threaded way,
you can consider using HTablePool.
Hope it helps
HJ
On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear Ted and Harsh,
>
> I am sorry I didn't keep the exceptions. It occurred many days ago. My
> current version is 0.92.
>
> Now "synchronized" is removed. Is it correct?
>
> I will test if such exceptions are raised. I will let you know.
>
> Thanks!
>
> Best wishes,
> Bing
>
>
> On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Bing:
> > Use pastebin.com instead of attaching exception report.
> >
> > What version of HBase are you using ?
> >
> > Thanks
> >
> >
> > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> What exceptions do you actually receive - can you send them here?
> >> Knowing that is key to addressing your issue.
> >>
> >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> >> > Dear all,
> >> >
> >> > When writing data into HBase, sometimes I got exceptions. I guess they
> >> > might be caused by concurrent writings. But I am not sure.
> >> >
> >> > My question is whether it is necessary to put "synchronized" before
> >> > the writing methods? The following lines are the sample code.
> >> >
> >> > I think the directive, synchronized, must lower the performance of
> >> > writing. Sometimes concurrent writing is needed in my system.
> >> >
> >> > Thanks so much!
> >> >
> >> > Best wishes,
> >> > Bing
> >> >
> >> > public synchronized void AddDomainNodeRanks(String domainKey, int
> >> > timingScale, Map<String, Double> nodeRankMap)
> >> > {
> >> > List<Put> puts = new ArrayList<Put>();
> >> > Put domainKeyPut;
> >> > Put timingScalePut;
> >> > Put nodeKeyPut;
> >> > Put rankPut;
> >> >
> >> > byte[] domainNodeRankRowKey;
> >> >
> >> > for (Map.Entry<String, Double> nodeRankEntry :
> >> > nodeRankMap.entrySet())
> >> > {
> >> > domainNodeRankRowKey > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> >> >
> >> > domainKeyPut = new Put(domainNodeRankRowKey);
> >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> >> > Bytes.toBytes(domainKey));
> >> > puts.add(domainKeyPut);
> >> >
> >> > timingScalePut = new Put(domainNodeRankRowKey);
> >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> >> > Bytes.toBytes(timingScale));
> >> > puts.add(timingScalePut);
> >> >
> >> > nodeKeyPut = new Put(domainNodeRankRowKey);
> >> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> >> > Bytes.toBytes(nodeRankEntry.getKey()));
> >> > puts.add(nodeKeyPut);
> >> >
> >> > rankPut = new Put(domainNodeRankRowKey);
> >> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> >> > Bytes.toBytes(nodeRankEntry.getValue()));
> >> > puts.add(rankPut);
> >> > }
> >> >
> >> > try
> >> > {
> >> > this.rankTable.put(puts);
> >> > }
> >> > catch (IOException e)
> >> > {
> >> > e.printStackTrace();
> >> > }
> >> > }
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
Not sure about your scenario but HTable class is not thread safe for
neither reads nor write.
If you consider writing/reading from a table in a multiple-threaded way,
you can consider using HTablePool.
Hope it helps
HJ
On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> Dear Ted and Harsh,
>
> I am sorry I didn't keep the exceptions. It occurred many days ago. My
> current version is 0.92.
>
> Now "synchronized" is removed. Is it correct?
>
> I will test if such exceptions are raised. I will let you know.
>
> Thanks!
>
> Best wishes,
> Bing
>
>
> On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Bing:
> > Use pastebin.com instead of attaching exception report.
> >
> > What version of HBase are you using ?
> >
> > Thanks
> >
> >
> > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> What exceptions do you actually receive - can you send them here?
> >> Knowing that is key to addressing your issue.
> >>
> >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> >> > Dear all,
> >> >
> >> > When writing data into HBase, sometimes I got exceptions. I guess they
> >> > might be caused by concurrent writings. But I am not sure.
> >> >
> >> > My question is whether it is necessary to put "synchronized" before
> >> > the writing methods? The following lines are the sample code.
> >> >
> >> > I think the directive, synchronized, must lower the performance of
> >> > writing. Sometimes concurrent writing is needed in my system.
> >> >
> >> > Thanks so much!
> >> >
> >> > Best wishes,
> >> > Bing
> >> >
> >> > public synchronized void AddDomainNodeRanks(String domainKey, int
> >> > timingScale, Map<String, Double> nodeRankMap)
> >> > {
> >> > List<Put> puts = new ArrayList<Put>();
> >> > Put domainKeyPut;
> >> > Put timingScalePut;
> >> > Put nodeKeyPut;
> >> > Put rankPut;
> >> >
> >> > byte[] domainNodeRankRowKey;
> >> >
> >> > for (Map.Entry<String, Double> nodeRankEntry :
> >> > nodeRankMap.entrySet())
> >> > {
> >> > domainNodeRankRowKey > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> >> >
> >> > domainKeyPut = new Put(domainNodeRankRowKey);
> >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> >> > Bytes.toBytes(domainKey));
> >> > puts.add(domainKeyPut);
> >> >
> >> > timingScalePut = new Put(domainNodeRankRowKey);
> >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> >> > Bytes.toBytes(timingScale));
> >> > puts.add(timingScalePut);
> >> >
> >> > nodeKeyPut = new Put(domainNodeRankRowKey);
> >> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> >> > Bytes.toBytes(nodeRankEntry.getKey()));
> >> > puts.add(nodeKeyPut);
> >> >
> >> > rankPut = new Put(domainNodeRankRowKey);
> >> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> >> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> >> > Bytes.toBytes(nodeRankEntry.getValue()));
> >> > puts.add(rankPut);
> >> > }
> >> >
> >> > try
> >> > {
> >> > this.rankTable.put(puts);
> >> > }
> >> > catch (IOException e)
> >> > {
> >> > e.printStackTrace();
> >> > }
> >> > }
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
Beware, HTablePool is not totally thread-safe as well:
https://issues.apache.org/jira/browse/HBASE-6651.
On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
> Hi, Bing,
>
> Not sure about your scenario but HTable class is not thread safe for
> neither reads nor write.
> If you consider writing/reading from a table in a multiple-threaded way,
> you can consider using HTablePool.
>
> Hope it helps
>
> HJ
>
>
> On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
> > Dear Ted and Harsh,
> >
> > I am sorry I didn't keep the exceptions. It occurred many days ago. My
> > current version is 0.92.
> >
> > Now "synchronized" is removed. Is it correct?
> >
> > I will test if such exceptions are raised. I will let you know.
> >
> > Thanks!
> >
> > Best wishes,
> > Bing
> >
> >
> > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > > Bing:
> > > Use pastebin.com instead of attaching exception report.
> > >
> > > What version of HBase are you using ?
> > >
> > > Thanks
> > >
> > >
> > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> > >>
> > >> What exceptions do you actually receive - can you send them here?
> > >> Knowing that is key to addressing your issue.
> > >>
> > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >> > Dear all,
> > >> >
> > >> > When writing data into HBase, sometimes I got exceptions. I guess
> they
> > >> > might be caused by concurrent writings. But I am not sure.
> > >> >
> > >> > My question is whether it is necessary to put "synchronized" before
> > >> > the writing methods? The following lines are the sample code.
> > >> >
> > >> > I think the directive, synchronized, must lower the performance of
> > >> > writing. Sometimes concurrent writing is needed in my system.
> > >> >
> > >> > Thanks so much!
> > >> >
> > >> > Best wishes,
> > >> > Bing
> > >> >
> > >> > public synchronized void AddDomainNodeRanks(String domainKey, int
> > >> > timingScale, Map<String, Double> nodeRankMap)
> > >> > {
> > >> > List<Put> puts = new ArrayList<Put>();
> > >> > Put domainKeyPut;
> > >> > Put timingScalePut;
> > >> > Put nodeKeyPut;
> > >> > Put rankPut;
> > >> >
> > >> > byte[] domainNodeRankRowKey;
> > >> >
> > >> > for (Map.Entry<String, Double> nodeRankEntry :
> > >> > nodeRankMap.entrySet())
> > >> > {
> > >> > domainNodeRankRowKey > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> > >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> > >> >
> > >> > domainKeyPut = new Put(domainNodeRankRowKey);
> > >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> > >> > Bytes.toBytes(domainKey));
> > >> > puts.add(domainKeyPut);
> > >> >
> > >> > timingScalePut = new Put(domainNodeRankRowKey);
> > >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> > >> > Bytes.toBytes(timingScale));
> > >> > puts.add(timingScalePut);
> > >> >
> > >> > nodeKeyPut = new Put(domainNodeRankRowKey);
> > >> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> > >> > Bytes.toBytes(nodeRankEntry.getKey()));
> > >> > puts.add(nodeKeyPut);
> > >> >
> > >> > rankPut = new Put(domainNodeRankRowKey);
> > >> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> > >> > Bytes.toBytes(nodeRankEntry.getValue()));
> > >> > puts.add(rankPut);
> > >> > }
> > >> >
> > >> > try
> > >> > {
> > >> > this.rankTable.put(puts);
> > >> > }
> > >> > catch (IOException e)
> > >> > {
> > >> > e.printStackTrace();
> > >> > }
> > >> > }
> > >>
> > >>
> > >>
> > >> --
> > >> Harsh J
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
https://issues.apache.org/jira/browse/HBASE-6651.
On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
> Hi, Bing,
>
> Not sure about your scenario but HTable class is not thread safe for
> neither reads nor write.
> If you consider writing/reading from a table in a multiple-threaded way,
> you can consider using HTablePool.
>
> Hope it helps
>
> HJ
>
>
> On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>
> > Dear Ted and Harsh,
> >
> > I am sorry I didn't keep the exceptions. It occurred many days ago. My
> > current version is 0.92.
> >
> > Now "synchronized" is removed. Is it correct?
> >
> > I will test if such exceptions are raised. I will let you know.
> >
> > Thanks!
> >
> > Best wishes,
> > Bing
> >
> >
> > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > > Bing:
> > > Use pastebin.com instead of attaching exception report.
> > >
> > > What version of HBase are you using ?
> > >
> > > Thanks
> > >
> > >
> > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> > >>
> > >> What exceptions do you actually receive - can you send them here?
> > >> Knowing that is key to addressing your issue.
> > >>
> > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> > >> > Dear all,
> > >> >
> > >> > When writing data into HBase, sometimes I got exceptions. I guess
> they
> > >> > might be caused by concurrent writings. But I am not sure.
> > >> >
> > >> > My question is whether it is necessary to put "synchronized" before
> > >> > the writing methods? The following lines are the sample code.
> > >> >
> > >> > I think the directive, synchronized, must lower the performance of
> > >> > writing. Sometimes concurrent writing is needed in my system.
> > >> >
> > >> > Thanks so much!
> > >> >
> > >> > Best wishes,
> > >> > Bing
> > >> >
> > >> > public synchronized void AddDomainNodeRanks(String domainKey, int
> > >> > timingScale, Map<String, Double> nodeRankMap)
> > >> > {
> > >> > List<Put> puts = new ArrayList<Put>();
> > >> > Put domainKeyPut;
> > >> > Put timingScalePut;
> > >> > Put nodeKeyPut;
> > >> > Put rankPut;
> > >> >
> > >> > byte[] domainNodeRankRowKey;
> > >> >
> > >> > for (Map.Entry<String, Double> nodeRankEntry :
> > >> > nodeRankMap.entrySet())
> > >> > {
> > >> > domainNodeRankRowKey > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> > >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> > >> >
> > >> > domainKeyPut = new Put(domainNodeRankRowKey);
> > >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> > >> > Bytes.toBytes(domainKey));
> > >> > puts.add(domainKeyPut);
> > >> >
> > >> > timingScalePut = new Put(domainNodeRankRowKey);
> > >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> > >> > Bytes.toBytes(timingScale));
> > >> > puts.add(timingScalePut);
> > >> >
> > >> > nodeKeyPut = new Put(domainNodeRankRowKey);
> > >> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> > >> > Bytes.toBytes(nodeRankEntry.getKey()));
> > >> > puts.add(nodeKeyPut);
> > >> >
> > >> > rankPut = new Put(domainNodeRankRowKey);
> > >> > rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > >> > RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
> > >> > Bytes.toBytes(nodeRankEntry.getValue()));
> > >> > puts.add(rankPut);
> > >> > }
> > >> >
> > >> > try
> > >> > {
> > >> > this.rankTable.put(puts);
> > >> > }
> > >> > catch (IOException e)
> > >> > {
> > >> > e.printStackTrace();
> > >> > }
> > >> > }
> > >>
> > >>
> > >>
> > >> --
> > >> Harsh J
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
Yes, HTable is not thread safe, and using synchronized around them could
work, but would be implementation dependent.
You can have one HTable per request at a reasonable cost since
https://issues.apache.org/jira/browse/HBASE-4805. It's seems to be
available in 0.92 as well.
Cheers,
Nicolas
On Mon, Feb 4, 2013 at 10:13 PM, Adrien Mogenet <[EMAIL PROTECTED]>wrote:
> Beware, HTablePool is not totally thread-safe as well:
> https://issues.apache.org/jira/browse/HBASE-6651.
>
>
> On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
>
> > Hi, Bing,
> >
> > Not sure about your scenario but HTable class is not thread safe for
> > neither reads nor write.
> > If you consider writing/reading from a table in a multiple-threaded way,
> > you can consider using HTablePool.
> >
> > Hope it helps
> >
> > HJ
> >
> >
> > On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >
> > > Dear Ted and Harsh,
> > >
> > > I am sorry I didn't keep the exceptions. It occurred many days ago. My
> > > current version is 0.92.
> > >
> > > Now "synchronized" is removed. Is it correct?
> > >
> > > I will test if such exceptions are raised. I will let you know.
> > >
> > > Thanks!
> > >
> > > Best wishes,
> > > Bing
> > >
> > >
> > > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > > > Bing:
> > > > Use pastebin.com instead of attaching exception report.
> > > >
> > > > What version of HBase are you using ?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> > > >>
> > > >> What exceptions do you actually receive - can you send them here?
> > > >> Knowing that is key to addressing your issue.
> > > >>
> > > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> > > >> > Dear all,
> > > >> >
> > > >> > When writing data into HBase, sometimes I got exceptions. I guess
> > they
> > > >> > might be caused by concurrent writings. But I am not sure.
> > > >> >
> > > >> > My question is whether it is necessary to put "synchronized"
> before
> > > >> > the writing methods? The following lines are the sample code.
> > > >> >
> > > >> > I think the directive, synchronized, must lower the performance of
> > > >> > writing. Sometimes concurrent writing is needed in my system.
> > > >> >
> > > >> > Thanks so much!
> > > >> >
> > > >> > Best wishes,
> > > >> > Bing
> > > >> >
> > > >> > public synchronized void AddDomainNodeRanks(String domainKey, int
> > > >> > timingScale, Map<String, Double> nodeRankMap)
> > > >> > {
> > > >> > List<Put> puts = new ArrayList<Put>();
> > > >> > Put domainKeyPut;
> > > >> > Put timingScalePut;
> > > >> > Put nodeKeyPut;
> > > >> > Put rankPut;
> > > >> >
> > > >> > byte[] domainNodeRankRowKey;
> > > >> >
> > > >> > for (Map.Entry<String, Double> nodeRankEntry :
> > > >> > nodeRankMap.entrySet())
> > > >> > {
> > > >> > domainNodeRankRowKey > > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> > > >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> > > >> >
> > > >> > domainKeyPut = new Put(domainNodeRankRowKey);
> > > >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > > >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> > > >> > Bytes.toBytes(domainKey));
> > > >> > puts.add(domainKeyPut);
> > > >> >
> > > >> > timingScalePut = new Put(domainNodeRankRowKey);
> > > >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > > >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> > > >> > Bytes.toBytes(timingScale));
> > > >> > puts.add(timingScalePut);
> > > >> >
> > > >> > nodeKeyPut = new Put(domainNodeRankRowKey);
> > > >> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > > >> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> > > >> > Bytes.toBytes(nodeRankEntry.getKey()));
> > > >> > puts.add(nodeKeyPut);
work, but would be implementation dependent.
You can have one HTable per request at a reasonable cost since
https://issues.apache.org/jira/browse/HBASE-4805. It's seems to be
available in 0.92 as well.
Cheers,
Nicolas
On Mon, Feb 4, 2013 at 10:13 PM, Adrien Mogenet <[EMAIL PROTECTED]>wrote:
> Beware, HTablePool is not totally thread-safe as well:
> https://issues.apache.org/jira/browse/HBASE-6651.
>
>
> On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
>
> > Hi, Bing,
> >
> > Not sure about your scenario but HTable class is not thread safe for
> > neither reads nor write.
> > If you consider writing/reading from a table in a multiple-threaded way,
> > you can consider using HTablePool.
> >
> > Hope it helps
> >
> > HJ
> >
> >
> > On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >
> > > Dear Ted and Harsh,
> > >
> > > I am sorry I didn't keep the exceptions. It occurred many days ago. My
> > > current version is 0.92.
> > >
> > > Now "synchronized" is removed. Is it correct?
> > >
> > > I will test if such exceptions are raised. I will let you know.
> > >
> > > Thanks!
> > >
> > > Best wishes,
> > > Bing
> > >
> > >
> > > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > > > Bing:
> > > > Use pastebin.com instead of attaching exception report.
> > > >
> > > > What version of HBase are you using ?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> > > >>
> > > >> What exceptions do you actually receive - can you send them here?
> > > >> Knowing that is key to addressing your issue.
> > > >>
> > > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> > > >> > Dear all,
> > > >> >
> > > >> > When writing data into HBase, sometimes I got exceptions. I guess
> > they
> > > >> > might be caused by concurrent writings. But I am not sure.
> > > >> >
> > > >> > My question is whether it is necessary to put "synchronized"
> before
> > > >> > the writing methods? The following lines are the sample code.
> > > >> >
> > > >> > I think the directive, synchronized, must lower the performance of
> > > >> > writing. Sometimes concurrent writing is needed in my system.
> > > >> >
> > > >> > Thanks so much!
> > > >> >
> > > >> > Best wishes,
> > > >> > Bing
> > > >> >
> > > >> > public synchronized void AddDomainNodeRanks(String domainKey, int
> > > >> > timingScale, Map<String, Double> nodeRankMap)
> > > >> > {
> > > >> > List<Put> puts = new ArrayList<Put>();
> > > >> > Put domainKeyPut;
> > > >> > Put timingScalePut;
> > > >> > Put nodeKeyPut;
> > > >> > Put rankPut;
> > > >> >
> > > >> > byte[] domainNodeRankRowKey;
> > > >> >
> > > >> > for (Map.Entry<String, Double> nodeRankEntry :
> > > >> > nodeRankMap.entrySet())
> > > >> > {
> > > >> > domainNodeRankRowKey > > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> > > >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
> > > >> >
> > > >> > domainKeyPut = new Put(domainNodeRankRowKey);
> > > >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > > >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
> > > >> > Bytes.toBytes(domainKey));
> > > >> > puts.add(domainKeyPut);
> > > >> >
> > > >> > timingScalePut = new Put(domainNodeRankRowKey);
> > > >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > > >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
> > > >> > Bytes.toBytes(timingScale));
> > > >> > puts.add(timingScalePut);
> > > >> >
> > > >> > nodeKeyPut = new Put(domainNodeRankRowKey);
> > > >> > nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
> > > >> > RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
> > > >> > Bytes.toBytes(nodeRankEntry.getKey()));
> > > >> > puts.add(nodeKeyPut);
Dear Nicolas,
If using synchronized is required, the performance must be too low,
right? Are there any other ways to minimize the synchronization
granularity?
Thanks so much!
Bing
On Tue, Feb 5, 2013 at 5:31 AM, Nicolas Liochon <[EMAIL PROTECTED]> wrote:
> Yes, HTable is not thread safe, and using synchronized around them could
> work, but would be implementation dependent.
> You can have one HTable per request at a reasonable cost since
> https://issues.apache.org/jira/browse/HBASE-4805. It's seems to be
> available in 0.92 as well.
>
> Cheers,
>
> Nicolas
>
>
> On Mon, Feb 4, 2013 at 10:13 PM, Adrien Mogenet <[EMAIL PROTECTED]>wrote:
>
>> Beware, HTablePool is not totally thread-safe as well:
>> https://issues.apache.org/jira/browse/HBASE-6651.
>>
>>
>> On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
>>
>> > Hi, Bing,
>> >
>> > Not sure about your scenario but HTable class is not thread safe for
>> > neither reads nor write.
>> > If you consider writing/reading from a table in a multiple-threaded way,
>> > you can consider using HTablePool.
>> >
>> > Hope it helps
>> >
>> > HJ
>> >
>> >
>> > On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>> >
>> > > Dear Ted and Harsh,
>> > >
>> > > I am sorry I didn't keep the exceptions. It occurred many days ago. My
>> > > current version is 0.92.
>> > >
>> > > Now "synchronized" is removed. Is it correct?
>> > >
>> > > I will test if such exceptions are raised. I will let you know.
>> > >
>> > > Thanks!
>> > >
>> > > Best wishes,
>> > > Bing
>> > >
>> > >
>> > > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > > > Bing:
>> > > > Use pastebin.com instead of attaching exception report.
>> > > >
>> > > > What version of HBase are you using ?
>> > > >
>> > > > Thanks
>> > > >
>> > > >
>> > > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> > > >>
>> > > >> What exceptions do you actually receive - can you send them here?
>> > > >> Knowing that is key to addressing your issue.
>> > > >>
>> > > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
>> > > >> > Dear all,
>> > > >> >
>> > > >> > When writing data into HBase, sometimes I got exceptions. I guess
>> > they
>> > > >> > might be caused by concurrent writings. But I am not sure.
>> > > >> >
>> > > >> > My question is whether it is necessary to put "synchronized"
>> before
>> > > >> > the writing methods? The following lines are the sample code.
>> > > >> >
>> > > >> > I think the directive, synchronized, must lower the performance of
>> > > >> > writing. Sometimes concurrent writing is needed in my system.
>> > > >> >
>> > > >> > Thanks so much!
>> > > >> >
>> > > >> > Best wishes,
>> > > >> > Bing
>> > > >> >
>> > > >> > public synchronized void AddDomainNodeRanks(String domainKey, int
>> > > >> > timingScale, Map<String, Double> nodeRankMap)
>> > > >> > {
>> > > >> > List<Put> puts = new ArrayList<Put>();
>> > > >> > Put domainKeyPut;
>> > > >> > Put timingScalePut;
>> > > >> > Put nodeKeyPut;
>> > > >> > Put rankPut;
>> > > >> >
>> > > >> > byte[] domainNodeRankRowKey;
>> > > >> >
>> > > >> > for (Map.Entry<String, Double> nodeRankEntry :
>> > > >> > nodeRankMap.entrySet())
>> > > >> > {
>> > > >> > domainNodeRankRowKey >> > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
>> > > >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
>> > > >> >
>> > > >> > domainKeyPut = new Put(domainNodeRankRowKey);
>> > > >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > > >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
>> > > >> > Bytes.toBytes(domainKey));
>> > > >> > puts.add(domainKeyPut);
>> > > >> >
>> > > >> > timingScalePut = new Put(domainNodeRankRowKey);
>> > > >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > > >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
If using synchronized is required, the performance must be too low,
right? Are there any other ways to minimize the synchronization
granularity?
Thanks so much!
Bing
On Tue, Feb 5, 2013 at 5:31 AM, Nicolas Liochon <[EMAIL PROTECTED]> wrote:
> Yes, HTable is not thread safe, and using synchronized around them could
> work, but would be implementation dependent.
> You can have one HTable per request at a reasonable cost since
> https://issues.apache.org/jira/browse/HBASE-4805. It's seems to be
> available in 0.92 as well.
>
> Cheers,
>
> Nicolas
>
>
> On Mon, Feb 4, 2013 at 10:13 PM, Adrien Mogenet <[EMAIL PROTECTED]>wrote:
>
>> Beware, HTablePool is not totally thread-safe as well:
>> https://issues.apache.org/jira/browse/HBASE-6651.
>>
>>
>> On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
>>
>> > Hi, Bing,
>> >
>> > Not sure about your scenario but HTable class is not thread safe for
>> > neither reads nor write.
>> > If you consider writing/reading from a table in a multiple-threaded way,
>> > you can consider using HTablePool.
>> >
>> > Hope it helps
>> >
>> > HJ
>> >
>> >
>> > On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
>> >
>> > > Dear Ted and Harsh,
>> > >
>> > > I am sorry I didn't keep the exceptions. It occurred many days ago. My
>> > > current version is 0.92.
>> > >
>> > > Now "synchronized" is removed. Is it correct?
>> > >
>> > > I will test if such exceptions are raised. I will let you know.
>> > >
>> > > Thanks!
>> > >
>> > > Best wishes,
>> > > Bing
>> > >
>> > >
>> > > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > > > Bing:
>> > > > Use pastebin.com instead of attaching exception report.
>> > > >
>> > > > What version of HBase are you using ?
>> > > >
>> > > > Thanks
>> > > >
>> > > >
>> > > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>> > > >>
>> > > >> What exceptions do you actually receive - can you send them here?
>> > > >> Knowing that is key to addressing your issue.
>> > > >>
>> > > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]> wrote:
>> > > >> > Dear all,
>> > > >> >
>> > > >> > When writing data into HBase, sometimes I got exceptions. I guess
>> > they
>> > > >> > might be caused by concurrent writings. But I am not sure.
>> > > >> >
>> > > >> > My question is whether it is necessary to put "synchronized"
>> before
>> > > >> > the writing methods? The following lines are the sample code.
>> > > >> >
>> > > >> > I think the directive, synchronized, must lower the performance of
>> > > >> > writing. Sometimes concurrent writing is needed in my system.
>> > > >> >
>> > > >> > Thanks so much!
>> > > >> >
>> > > >> > Best wishes,
>> > > >> > Bing
>> > > >> >
>> > > >> > public synchronized void AddDomainNodeRanks(String domainKey, int
>> > > >> > timingScale, Map<String, Double> nodeRankMap)
>> > > >> > {
>> > > >> > List<Put> puts = new ArrayList<Put>();
>> > > >> > Put domainKeyPut;
>> > > >> > Put timingScalePut;
>> > > >> > Put nodeKeyPut;
>> > > >> > Put rankPut;
>> > > >> >
>> > > >> > byte[] domainNodeRankRowKey;
>> > > >> >
>> > > >> > for (Map.Entry<String, Double> nodeRankEntry :
>> > > >> > nodeRankMap.entrySet())
>> > > >> > {
>> > > >> > domainNodeRankRowKey >> > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
>> > > >> > Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
>> > > >> >
>> > > >> > domainKeyPut = new Put(domainNodeRankRowKey);
>> > > >> > domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > > >> > RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
>> > > >> > Bytes.toBytes(domainKey));
>> > > >> > puts.add(domainKeyPut);
>> > > >> >
>> > > >> > timingScalePut = new Put(domainNodeRankRowKey);
>> > > >> > timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> > > >> > RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
You don't have to use synchronized, just create multiple instances of
HTable and you will be fine.
Le 4 févr. 2013 23:40, "Bing Li" <[EMAIL PROTECTED]> a écrit :
> Dear Nicolas,
>
> If using synchronized is required, the performance must be too low,
> right? Are there any other ways to minimize the synchronization
> granularity?
>
> Thanks so much!
> Bing
>
> On Tue, Feb 5, 2013 at 5:31 AM, Nicolas Liochon <[EMAIL PROTECTED]> wrote:
> > Yes, HTable is not thread safe, and using synchronized around them could
> > work, but would be implementation dependent.
> > You can have one HTable per request at a reasonable cost since
> > https://issues.apache.org/jira/browse/HBASE-4805. It's seems to be
> > available in 0.92 as well.
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> > On Mon, Feb 4, 2013 at 10:13 PM, Adrien Mogenet <
> [EMAIL PROTECTED]>wrote:
> >
> >> Beware, HTablePool is not totally thread-safe as well:
> >> https://issues.apache.org/jira/browse/HBASE-6651.
> >>
> >>
> >> On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
> >>
> >> > Hi, Bing,
> >> >
> >> > Not sure about your scenario but HTable class is not thread safe for
> >> > neither reads nor write.
> >> > If you consider writing/reading from a table in a multiple-threaded
> way,
> >> > you can consider using HTablePool.
> >> >
> >> > Hope it helps
> >> >
> >> > HJ
> >> >
> >> >
> >> > On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > Dear Ted and Harsh,
> >> > >
> >> > > I am sorry I didn't keep the exceptions. It occurred many days ago.
> My
> >> > > current version is 0.92.
> >> > >
> >> > > Now "synchronized" is removed. Is it correct?
> >> > >
> >> > > I will test if such exceptions are raised. I will let you know.
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Best wishes,
> >> > > Bing
> >> > >
> >> > >
> >> > > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >> > > > Bing:
> >> > > > Use pastebin.com instead of attaching exception report.
> >> > > >
> >> > > > What version of HBase are you using ?
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > >
> >> > > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]>
> wrote:
> >> > > >>
> >> > > >> What exceptions do you actually receive - can you send them here?
> >> > > >> Knowing that is key to addressing your issue.
> >> > > >>
> >> > > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]>
> wrote:
> >> > > >> > Dear all,
> >> > > >> >
> >> > > >> > When writing data into HBase, sometimes I got exceptions. I
> guess
> >> > they
> >> > > >> > might be caused by concurrent writings. But I am not sure.
> >> > > >> >
> >> > > >> > My question is whether it is necessary to put "synchronized"
> >> before
> >> > > >> > the writing methods? The following lines are the sample code.
> >> > > >> >
> >> > > >> > I think the directive, synchronized, must lower the
> performance of
> >> > > >> > writing. Sometimes concurrent writing is needed in my system.
> >> > > >> >
> >> > > >> > Thanks so much!
> >> > > >> >
> >> > > >> > Best wishes,
> >> > > >> > Bing
> >> > > >> >
> >> > > >> > public synchronized void AddDomainNodeRanks(String domainKey,
> int
> >> > > >> > timingScale, Map<String, Double> nodeRankMap)
> >> > > >> > {
> >> > > >> > List<Put> puts = new ArrayList<Put>();
> >> > > >> > Put domainKeyPut;
> >> > > >> > Put timingScalePut;
> >> > > >> > Put nodeKeyPut;
> >> > > >> > Put rankPut;
> >> > > >> >
> >> > > >> > byte[] domainNodeRankRowKey;
> >> > > >> >
> >> > > >> > for (Map.Entry<String, Double> nodeRankEntry :
> >> > > >> > nodeRankMap.entrySet())
> >> > > >> > {
> >> > > >> > domainNodeRankRowKey > >> > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> >> > > >> > Tools.GetAHash(domainKey + timingScale +
> nodeRankEntry.getKey()));
> >> > > >> >
> >> > > >> > domainKeyPut = new Put(domainNodeRankRowKey);
> >> > > >> >
> domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
HTable and you will be fine.
Le 4 févr. 2013 23:40, "Bing Li" <[EMAIL PROTECTED]> a écrit :
> Dear Nicolas,
>
> If using synchronized is required, the performance must be too low,
> right? Are there any other ways to minimize the synchronization
> granularity?
>
> Thanks so much!
> Bing
>
> On Tue, Feb 5, 2013 at 5:31 AM, Nicolas Liochon <[EMAIL PROTECTED]> wrote:
> > Yes, HTable is not thread safe, and using synchronized around them could
> > work, but would be implementation dependent.
> > You can have one HTable per request at a reasonable cost since
> > https://issues.apache.org/jira/browse/HBASE-4805. It's seems to be
> > available in 0.92 as well.
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> > On Mon, Feb 4, 2013 at 10:13 PM, Adrien Mogenet <
> [EMAIL PROTECTED]>wrote:
> >
> >> Beware, HTablePool is not totally thread-safe as well:
> >> https://issues.apache.org/jira/browse/HBASE-6651.
> >>
> >>
> >> On Mon, Feb 4, 2013 at 9:42 PM, Haijia Zhou <[EMAIL PROTECTED]> wrote:
> >>
> >> > Hi, Bing,
> >> >
> >> > Not sure about your scenario but HTable class is not thread safe for
> >> > neither reads nor write.
> >> > If you consider writing/reading from a table in a multiple-threaded
> way,
> >> > you can consider using HTablePool.
> >> >
> >> > Hope it helps
> >> >
> >> > HJ
> >> >
> >> >
> >> > On Mon, Feb 4, 2013 at 3:32 PM, Bing Li <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > Dear Ted and Harsh,
> >> > >
> >> > > I am sorry I didn't keep the exceptions. It occurred many days ago.
> My
> >> > > current version is 0.92.
> >> > >
> >> > > Now "synchronized" is removed. Is it correct?
> >> > >
> >> > > I will test if such exceptions are raised. I will let you know.
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Best wishes,
> >> > > Bing
> >> > >
> >> > >
> >> > > On Tue, Feb 5, 2013 at 4:25 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >> > > > Bing:
> >> > > > Use pastebin.com instead of attaching exception report.
> >> > > >
> >> > > > What version of HBase are you using ?
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > >
> >> > > > On Mon, Feb 4, 2013 at 12:21 PM, Harsh J <[EMAIL PROTECTED]>
> wrote:
> >> > > >>
> >> > > >> What exceptions do you actually receive - can you send them here?
> >> > > >> Knowing that is key to addressing your issue.
> >> > > >>
> >> > > >> On Tue, Feb 5, 2013 at 1:50 AM, Bing Li <[EMAIL PROTECTED]>
> wrote:
> >> > > >> > Dear all,
> >> > > >> >
> >> > > >> > When writing data into HBase, sometimes I got exceptions. I
> guess
> >> > they
> >> > > >> > might be caused by concurrent writings. But I am not sure.
> >> > > >> >
> >> > > >> > My question is whether it is necessary to put "synchronized"
> >> before
> >> > > >> > the writing methods? The following lines are the sample code.
> >> > > >> >
> >> > > >> > I think the directive, synchronized, must lower the
> performance of
> >> > > >> > writing. Sometimes concurrent writing is needed in my system.
> >> > > >> >
> >> > > >> > Thanks so much!
> >> > > >> >
> >> > > >> > Best wishes,
> >> > > >> > Bing
> >> > > >> >
> >> > > >> > public synchronized void AddDomainNodeRanks(String domainKey,
> int
> >> > > >> > timingScale, Map<String, Double> nodeRankMap)
> >> > > >> > {
> >> > > >> > List<Put> puts = new ArrayList<Put>();
> >> > > >> > Put domainKeyPut;
> >> > > >> > Put timingScalePut;
> >> > > >> > Put nodeKeyPut;
> >> > > >> > Put rankPut;
> >> > > >> >
> >> > > >> > byte[] domainNodeRankRowKey;
> >> > > >> >
> >> > > >> > for (Map.Entry<String, Double> nodeRankEntry :
> >> > > >> > nodeRankMap.entrySet())
> >> > > >> > {
> >> > > >> > domainNodeRankRowKey > >> > > >> > Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
> >> > > >> > Tools.GetAHash(domainKey + timingScale +
> nodeRankEntry.getKey()));
> >> > > >> >
> >> > > >> > domainKeyPut = new Put(domainNodeRankRowKey);
> >> > > >> >
> domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
Dear all,
After "synchronized" is removed from the method of writing, I get the
following exceptions when reading. Before the removal, no such
exceptions.
Could you help me how to solve it?
Thanks so much!
Best wishes,
Bing
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
[java] WARNING: Unexpected exception receiving call responses
[java] java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
[java] WARNING: Ignore, probably already closed
[java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
[java] at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
[java] at $Proxy6.close(Unknown Source)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
[java] at
com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
[java] at
com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[java] at java.lang.Thread.run(Thread.java:662)
[java] Caused by: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
[java] Caused by: java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
The code that causes the exceptions is as follows.
public Set<String> LoadNodeGroupNodeRankRowKeys(String
hostNodeKey, String groupKey, int timingScale)
{
List<Filter> nodeGroupFilterList = new ArrayList<Filter>();
SingleColumnValueFilter hostNodeKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_HOST_NODE_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(hostNodeKey));
hostNodeKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(hostNodeKeyFilter);
SingleColumnValueFilter groupKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_GROUP_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(groupKey));
groupKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(groupKeyFilter);
SingleColumnValueFilter timingScaleFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_TIMING_SCALE_COLUMN,
CompareFilter.CompareOp.EQUAL, new
BinaryComparator(Bytes.toBytes(timingScale)));
timingScaleFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(timingScaleFilter);
FilterList nodeGroupFilter = new
FilterList(nodeGroupFilterList);
Scan scan = new Scan();
scan.setFilter(nodeGroupFilter);
scan.setCaching(Parameters.CACHING_SIZE);
scan.setBatch(Parameters.BATCHING_SIZE);
Set<String> rowKeySet = Sets.newHashSet();
try
{
ResultScanner scanner = this.rankTable.getScanner(scan);
for (Result result : scanner) //
<---- EXCEPTIONS are raised at this line.
{
for (KeyValue kv : result.raw())
{
rowKeySet.add(Bytes.toString(kv.getRow()));
break;
}
}
scanner.close();
}
catch (IOException e)
{
e.printStackTrace();
}
return rowKeySet;
}
On Tue, Feb 5, 2013 at 4:20 AM, Bing Li <[EMAIL PROTECTED]> wrote:
After "synchronized" is removed from the method of writing, I get the
following exceptions when reading. Before the removal, no such
exceptions.
Could you help me how to solve it?
Thanks so much!
Best wishes,
Bing
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
[java] WARNING: Unexpected exception receiving call responses
[java] java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
[java] WARNING: Ignore, probably already closed
[java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
[java] at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
[java] at $Proxy6.close(Unknown Source)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
[java] at
com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
[java] at
com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[java] at java.lang.Thread.run(Thread.java:662)
[java] Caused by: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
[java] Caused by: java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
The code that causes the exceptions is as follows.
public Set<String> LoadNodeGroupNodeRankRowKeys(String
hostNodeKey, String groupKey, int timingScale)
{
List<Filter> nodeGroupFilterList = new ArrayList<Filter>();
SingleColumnValueFilter hostNodeKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_HOST_NODE_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(hostNodeKey));
hostNodeKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(hostNodeKeyFilter);
SingleColumnValueFilter groupKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_GROUP_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(groupKey));
groupKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(groupKeyFilter);
SingleColumnValueFilter timingScaleFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_TIMING_SCALE_COLUMN,
CompareFilter.CompareOp.EQUAL, new
BinaryComparator(Bytes.toBytes(timingScale)));
timingScaleFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(timingScaleFilter);
FilterList nodeGroupFilter = new
FilterList(nodeGroupFilterList);
Scan scan = new Scan();
scan.setFilter(nodeGroupFilter);
scan.setCaching(Parameters.CACHING_SIZE);
scan.setBatch(Parameters.BATCHING_SIZE);
Set<String> rowKeySet = Sets.newHashSet();
try
{
ResultScanner scanner = this.rankTable.getScanner(scan);
for (Result result : scanner) //
<---- EXCEPTIONS are raised at this line.
{
for (KeyValue kv : result.raw())
{
rowKeySet.add(Bytes.toString(kv.getRow()));
break;
}
}
scanner.close();
}
catch (IOException e)
{
e.printStackTrace();
}
return rowKeySet;
}
On Tue, Feb 5, 2013 at 4:20 AM, Bing Li <[EMAIL PROTECTED]> wrote:
Are you sharing this.rankTable between threads? HTable is not thread safe.
-- Lars
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
Sent: Tuesday, February 5, 2013 8:54 AM
Subject: Re: Is "synchronized" required?
Dear all,
After "synchronized" is removed from the method of writing, I get the
following exceptions when reading. Before the removal, no such
exceptions.
Could you help me how to solve it?
Thanks so much!
Best wishes,
Bing
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
[java] WARNING: Unexpected exception receiving call responses
[java] java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
[java] WARNING: Ignore, probably already closed
[java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
[java] at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
[java] at $Proxy6.close(Unknown Source)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
[java] at
com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
[java] at
com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[java] at java.lang.Thread.run(Thread.java:662)
[java] Caused by: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
[java] Caused by: java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
The code that causes the exceptions is as follows.
public Set<String> LoadNodeGroupNodeRankRowKeys(String
hostNodeKey, String groupKey, int timingScale)
{
List<Filter> nodeGroupFilterList = new ArrayList<Filter>();
SingleColumnValueFilter hostNodeKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_HOST_NODE_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(hostNodeKey));
hostNodeKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(hostNodeKeyFilter);
SingleColumnValueFilter groupKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_GROUP_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(groupKey));
groupKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(groupKeyFilter);
SingleColumnValueFilter timingScaleFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_TIMING_SCALE_COLUMN,
CompareFilter.CompareOp.EQUAL, new
BinaryComparator(Bytes.toBytes(timingScale)));
timingScaleFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(timingScaleFilter);
FilterList nodeGroupFilter = new
FilterList(nodeGroupFilterList);
Scan scan = new Scan();
scan.setFilter(nodeGroupFilter);
scan.setCaching(Parameters.CACHING_SIZE);
scan.setBatch(Parameters.BATCHING_SIZE);
Set<String> rowKeySet = Sets.newHashSet();
try
{
ResultScanner scanner = this.rankTable.getScanner(scan);
for (Result result : scanner) //
<---- EXCEPTIONS are raised at this line.
{
for (KeyValue kv : result.raw())
{
rowKeySet.add(Bytes.toString(kv.getRow()));
break;
}
}
scanner.close();
-- Lars
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
Sent: Tuesday, February 5, 2013 8:54 AM
Subject: Re: Is "synchronized" required?
Dear all,
After "synchronized" is removed from the method of writing, I get the
following exceptions when reading. Before the removal, no such
exceptions.
Could you help me how to solve it?
Thanks so much!
Best wishes,
Bing
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
[java] WARNING: Unexpected exception receiving call responses
[java] java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
[java] Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
[java] WARNING: Ignore, probably already closed
[java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
[java] at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
[java] at $Proxy6.close(Unknown Source)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
[java] at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
[java] at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
[java] at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
[java] at
com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
[java] at
com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[java] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[java] at java.lang.Thread.run(Thread.java:662)
[java] Caused by: java.io.IOException: Unexpected exception
receiving call responses
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
[java] Caused by: java.lang.NullPointerException
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
[java] at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
[java] at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
The code that causes the exceptions is as follows.
public Set<String> LoadNodeGroupNodeRankRowKeys(String
hostNodeKey, String groupKey, int timingScale)
{
List<Filter> nodeGroupFilterList = new ArrayList<Filter>();
SingleColumnValueFilter hostNodeKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_HOST_NODE_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(hostNodeKey));
hostNodeKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(hostNodeKeyFilter);
SingleColumnValueFilter groupKeyFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_GROUP_KEY_COLUMN,
CompareFilter.CompareOp.EQUAL, new SubstringComparator(groupKey));
groupKeyFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(groupKeyFilter);
SingleColumnValueFilter timingScaleFilter = new
SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
RankStructure.NODE_GROUP_NODE_RANK_TIMING_SCALE_COLUMN,
CompareFilter.CompareOp.EQUAL, new
BinaryComparator(Bytes.toBytes(timingScale)));
timingScaleFilter.setFilterIfMissing(true);
nodeGroupFilterList.add(timingScaleFilter);
FilterList nodeGroupFilter = new
FilterList(nodeGroupFilterList);
Scan scan = new Scan();
scan.setFilter(nodeGroupFilter);
scan.setCaching(Parameters.CACHING_SIZE);
scan.setBatch(Parameters.BATCHING_SIZE);
Set<String> rowKeySet = Sets.newHashSet();
try
{
ResultScanner scanner = this.rankTable.getScanner(scan);
for (Result result : scanner) //
<---- EXCEPTIONS are raised at this line.
{
for (KeyValue kv : result.raw())
{
rowKeySet.add(Bytes.toString(kv.getRow()));
break;
}
}
scanner.close();
Lars,
I found that at least the exceptions have nothing to do with shared HTable.
To save the resources, I designed a pool for the classes that write
and read from HBase. The primary resources consumed in the classes are
HTable. The pool has some bugs.
My question is whether it is necessary to design such a pool? Is it
fine to create a instance of HTable for each thread?
I noticed that HBase has a class, HTablePool. Maybe the pool I
designed is NOT required?
Thanks so much!
Best wishes!
Bing
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Are you sharing this.rankTable between threads? HTable is not thread safe.
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 8:54 AM
> Subject: Re: Is "synchronized" required?
>
> Dear all,
>
> After "synchronized" is removed from the method of writing, I get the
> following exceptions when reading. Before the removal, no such
> exceptions.
>
> Could you help me how to solve it?
>
> Thanks so much!
>
> Best wishes,
> Bing
>
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
> [java] WARNING: Unexpected exception receiving call responses
> [java] java.lang.NullPointerException
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.client.ScannerCallable close
> [java] WARNING: Ignore, probably already closed
> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
> failed on local exception: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
> [java] at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> [java] at $Proxy6.close(Unknown Source)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
> [java] at
> com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
> [java] at
> com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
> [java] at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [java] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [java] at java.lang.Thread.run(Thread.java:662)
> [java] Caused by: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
> [java] Caused by: java.lang.NullPointerException
I found that at least the exceptions have nothing to do with shared HTable.
To save the resources, I designed a pool for the classes that write
and read from HBase. The primary resources consumed in the classes are
HTable. The pool has some bugs.
My question is whether it is necessary to design such a pool? Is it
fine to create a instance of HTable for each thread?
I noticed that HBase has a class, HTablePool. Maybe the pool I
designed is NOT required?
Thanks so much!
Best wishes!
Bing
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Are you sharing this.rankTable between threads? HTable is not thread safe.
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 8:54 AM
> Subject: Re: Is "synchronized" required?
>
> Dear all,
>
> After "synchronized" is removed from the method of writing, I get the
> following exceptions when reading. Before the removal, no such
> exceptions.
>
> Could you help me how to solve it?
>
> Thanks so much!
>
> Best wishes,
> Bing
>
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
> [java] WARNING: Unexpected exception receiving call responses
> [java] java.lang.NullPointerException
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.client.ScannerCallable close
> [java] WARNING: Ignore, probably already closed
> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
> failed on local exception: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
> [java] at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> [java] at $Proxy6.close(Unknown Source)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
> [java] at
> com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
> [java] at
> com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
> [java] at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [java] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [java] at java.lang.Thread.run(Thread.java:662)
> [java] Caused by: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
> [java] Caused by: java.lang.NullPointerException
Don't use a pool at all.
With HBASE-4805 (https://issues.apache.org/jira/browse/HBASE-4805) you can precreate an HConnection and ExecutorService and then make HTable cheaply on demand every time you need one.
Checkout HConnectionManager.createConnection(...) and the HTable constructors.
I need to document this somewhere.
-- Lars
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Tuesday, February 5, 2013 10:36 PM
Subject: Re: Is "synchronized" required?
Lars,
I found that at least the exceptions have nothing to do with shared HTable.
To save the resources, I designed a pool for the classes that write
and read from HBase. The primary resources consumed in the classes are
HTable. The pool has some bugs.
My question is whether it is necessary to design such a pool? Is it
fine to create a instance of HTable for each thread?
I noticed that HBase has a class, HTablePool. Maybe the pool I
designed is NOT required?
Thanks so much!
Best wishes!
Bing
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Are you sharing this.rankTable between threads? HTable is not thread safe.
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 8:54 AM
> Subject: Re: Is "synchronized" required?
>
> Dear all,
>
> After "synchronized" is removed from the method of writing, I get the
> following exceptions when reading. Before the removal, no such
> exceptions.
>
> Could you help me how to solve it?
>
> Thanks so much!
>
> Best wishes,
> Bing
>
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
> [java] WARNING: Unexpected exception receiving call responses
> [java] java.lang.NullPointerException
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.client.ScannerCallable close
> [java] WARNING: Ignore, probably already closed
> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
> failed on local exception: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
> [java] at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> [java] at $Proxy6.close(Unknown Source)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
> [java] at
> com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
> [java] at
With HBASE-4805 (https://issues.apache.org/jira/browse/HBASE-4805) you can precreate an HConnection and ExecutorService and then make HTable cheaply on demand every time you need one.
Checkout HConnectionManager.createConnection(...) and the HTable constructors.
I need to document this somewhere.
-- Lars
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Tuesday, February 5, 2013 10:36 PM
Subject: Re: Is "synchronized" required?
Lars,
I found that at least the exceptions have nothing to do with shared HTable.
To save the resources, I designed a pool for the classes that write
and read from HBase. The primary resources consumed in the classes are
HTable. The pool has some bugs.
My question is whether it is necessary to design such a pool? Is it
fine to create a instance of HTable for each thread?
I noticed that HBase has a class, HTablePool. Maybe the pool I
designed is NOT required?
Thanks so much!
Best wishes!
Bing
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Are you sharing this.rankTable between threads? HTable is not thread safe.
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 8:54 AM
> Subject: Re: Is "synchronized" required?
>
> Dear all,
>
> After "synchronized" is removed from the method of writing, I get the
> following exceptions when reading. Before the removal, no such
> exceptions.
>
> Could you help me how to solve it?
>
> Thanks so much!
>
> Best wishes,
> Bing
>
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
> [java] WARNING: Unexpected exception receiving call responses
> [java] java.lang.NullPointerException
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.client.ScannerCallable close
> [java] WARNING: Ignore, probably already closed
> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
> failed on local exception: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
> [java] at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> [java] at $Proxy6.close(Unknown Source)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
> [java] at
> com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
> [java] at
I probably don't know your application enough to make an accurate answer,
but you could have a look at asynchbase [
https://github.com/OpenTSDB/asynchbase] if you have thread-safety issues
and feel the need to control your resources over your threads.
On Wed, Feb 6, 2013 at 7:36 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> Lars,
>
> I found that at least the exceptions have nothing to do with shared HTable.
>
> To save the resources, I designed a pool for the classes that write
> and read from HBase. The primary resources consumed in the classes are
> HTable. The pool has some bugs.
>
> My question is whether it is necessary to design such a pool? Is it
> fine to create a instance of HTable for each thread?
>
> I noticed that HBase has a class, HTablePool. Maybe the pool I
> designed is NOT required?
>
> Thanks so much!
>
> Best wishes!
> Bing
>
> On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> > Are you sharing this.rankTable between threads? HTable is not thread
> safe.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Bing Li <[EMAIL PROTECTED]>
> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user
> <[EMAIL PROTECTED]>
> > Sent: Tuesday, February 5, 2013 8:54 AM
> > Subject: Re: Is "synchronized" required?
> >
> > Dear all,
> >
> > After "synchronized" is removed from the method of writing, I get the
> > following exceptions when reading. Before the removal, no such
> > exceptions.
> >
> > Could you help me how to solve it?
> >
> > Thanks so much!
> >
> > Best wishes,
> > Bing
> >
> > [java] Feb 6, 2013 12:21:31 AM
> > org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
> > [java] WARNING: Unexpected exception receiving call responses
> > [java] java.lang.NullPointerException
> > [java] at
> >
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> > [java] at
> >
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
> > [java] Feb 6, 2013 12:21:31 AM
> > org.apache.hadoop.hbase.client.ScannerCallable close
> > [java] WARNING: Ignore, probably already closed
> > [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
> > failed on local exception: java.io.IOException: Unexpected exception
> > receiving call responses
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
> > [java] at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> > [java] at $Proxy6.close(Unknown Source)
> > [java] at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
> > [java] at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
> > [java] at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
> > [java] at
> >
> com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
> > [java] at
> >
> com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
but you could have a look at asynchbase [
https://github.com/OpenTSDB/asynchbase] if you have thread-safety issues
and feel the need to control your resources over your threads.
On Wed, Feb 6, 2013 at 7:36 AM, Bing Li <[EMAIL PROTECTED]> wrote:
> Lars,
>
> I found that at least the exceptions have nothing to do with shared HTable.
>
> To save the resources, I designed a pool for the classes that write
> and read from HBase. The primary resources consumed in the classes are
> HTable. The pool has some bugs.
>
> My question is whether it is necessary to design such a pool? Is it
> fine to create a instance of HTable for each thread?
>
> I noticed that HBase has a class, HTablePool. Maybe the pool I
> designed is NOT required?
>
> Thanks so much!
>
> Best wishes!
> Bing
>
> On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> > Are you sharing this.rankTable between threads? HTable is not thread
> safe.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Bing Li <[EMAIL PROTECTED]>
> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user
> <[EMAIL PROTECTED]>
> > Sent: Tuesday, February 5, 2013 8:54 AM
> > Subject: Re: Is "synchronized" required?
> >
> > Dear all,
> >
> > After "synchronized" is removed from the method of writing, I get the
> > following exceptions when reading. Before the removal, no such
> > exceptions.
> >
> > Could you help me how to solve it?
> >
> > Thanks so much!
> >
> > Best wishes,
> > Bing
> >
> > [java] Feb 6, 2013 12:21:31 AM
> > org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
> > [java] WARNING: Unexpected exception receiving call responses
> > [java] java.lang.NullPointerException
> > [java] at
> >
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> > [java] at
> >
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
> > [java] Feb 6, 2013 12:21:31 AM
> > org.apache.hadoop.hbase.client.ScannerCallable close
> > [java] WARNING: Ignore, probably already closed
> > [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
> > failed on local exception: java.io.IOException: Unexpected exception
> > receiving call responses
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
> > [java] at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
> > [java] at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> > [java] at $Proxy6.close(Unknown Source)
> > [java] at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
> > [java] at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
> > [java] at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
> > [java] at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
> > [java] at
> >
> com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
> > [java] at
> >
> com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
Dear Lars,
I am now running HBase in the pseudo-distributed mode. The updated
HTable constructor also works?
Thanks so much!
Bing
On Wed, Feb 6, 2013 at 3:44 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Don't use a pool at all.
> With HBASE-4805 (https://issues.apache.org/jira/browse/HBASE-4805) you can precreate an HConnection and ExecutorService and then make HTable cheaply on demand every time you need one.
>
> Checkout HConnectionManager.createConnection(...) and the HTable constructors.
>
> I need to document this somewhere.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 10:36 PM
> Subject: Re: Is "synchronized" required?
>
> Lars,
>
> I found that at least the exceptions have nothing to do with shared HTable.
>
> To save the resources, I designed a pool for the classes that write
> and read from HBase. The primary resources consumed in the classes are
> HTable. The pool has some bugs.
>
> My question is whether it is necessary to design such a pool? Is it
> fine to create a instance of HTable for each thread?
>
> I noticed that HBase has a class, HTablePool. Maybe the pool I
> designed is NOT required?
>
> Thanks so much!
>
> Best wishes!
> Bing
>
> On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> Are you sharing this.rankTable between threads? HTable is not thread safe.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Bing Li <[EMAIL PROTECTED]>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
>> Sent: Tuesday, February 5, 2013 8:54 AM
>> Subject: Re: Is "synchronized" required?
>>
>> Dear all,
>>
>> After "synchronized" is removed from the method of writing, I get the
>> following exceptions when reading. Before the removal, no such
>> exceptions.
>>
>> Could you help me how to solve it?
>>
>> Thanks so much!
>>
>> Best wishes,
>> Bing
>>
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
>> [java] WARNING: Unexpected exception receiving call responses
>> [java] java.lang.NullPointerException
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.client.ScannerCallable close
>> [java] WARNING: Ignore, probably already closed
>> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
>> failed on local exception: java.io.IOException: Unexpected exception
>> receiving call responses
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
>> [java] at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
>> [java] at $Proxy6.close(Unknown Source)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
>> [java] at
>> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
>> [java] at
I am now running HBase in the pseudo-distributed mode. The updated
HTable constructor also works?
Thanks so much!
Bing
On Wed, Feb 6, 2013 at 3:44 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Don't use a pool at all.
> With HBASE-4805 (https://issues.apache.org/jira/browse/HBASE-4805) you can precreate an HConnection and ExecutorService and then make HTable cheaply on demand every time you need one.
>
> Checkout HConnectionManager.createConnection(...) and the HTable constructors.
>
> I need to document this somewhere.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 10:36 PM
> Subject: Re: Is "synchronized" required?
>
> Lars,
>
> I found that at least the exceptions have nothing to do with shared HTable.
>
> To save the resources, I designed a pool for the classes that write
> and read from HBase. The primary resources consumed in the classes are
> HTable. The pool has some bugs.
>
> My question is whether it is necessary to design such a pool? Is it
> fine to create a instance of HTable for each thread?
>
> I noticed that HBase has a class, HTablePool. Maybe the pool I
> designed is NOT required?
>
> Thanks so much!
>
> Best wishes!
> Bing
>
> On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> Are you sharing this.rankTable between threads? HTable is not thread safe.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Bing Li <[EMAIL PROTECTED]>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
>> Sent: Tuesday, February 5, 2013 8:54 AM
>> Subject: Re: Is "synchronized" required?
>>
>> Dear all,
>>
>> After "synchronized" is removed from the method of writing, I get the
>> following exceptions when reading. Before the removal, no such
>> exceptions.
>>
>> Could you help me how to solve it?
>>
>> Thanks so much!
>>
>> Best wishes,
>> Bing
>>
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
>> [java] WARNING: Unexpected exception receiving call responses
>> [java] java.lang.NullPointerException
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.client.ScannerCallable close
>> [java] WARNING: Ignore, probably already closed
>> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
>> failed on local exception: java.io.IOException: Unexpected exception
>> receiving call responses
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
>> [java] at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
>> [java] at $Proxy6.close(Unknown Source)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
>> [java] at
>> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
>> [java] at
Yep.
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>
Sent: Wednesday, February 6, 2013 2:31 AM
Subject: Re: Is "synchronized" required?
Dear Lars,
I am now running HBase in the pseudo-distributed mode. The updated
HTable constructor also works?
Thanks so much!
Bing
On Wed, Feb 6, 2013 at 3:44 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Don't use a pool at all.
> With HBASE-4805 (https://issues.apache.org/jira/browse/HBASE-4805) you can precreate an HConnection and ExecutorService and then make HTable cheaply on demand every time you need one.
>
> Checkout HConnectionManager.createConnection(...) and the HTable constructors.
>
> I need to document this somewhere.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 10:36 PM
> Subject: Re: Is "synchronized" required?
>
> Lars,
>
> I found that at least the exceptions have nothing to do with shared HTable.
>
> To save the resources, I designed a pool for the classes that write
> and read from HBase. The primary resources consumed in the classes are
> HTable. The pool has some bugs.
>
> My question is whether it is necessary to design such a pool? Is it
> fine to create a instance of HTable for each thread?
>
> I noticed that HBase has a class, HTablePool. Maybe the pool I
> designed is NOT required?
>
> Thanks so much!
>
> Best wishes!
> Bing
>
> On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> Are you sharing this.rankTable between threads? HTable is not thread safe.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Bing Li <[EMAIL PROTECTED]>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
>> Sent: Tuesday, February 5, 2013 8:54 AM
>> Subject: Re: Is "synchronized" required?
>>
>> Dear all,
>>
>> After "synchronized" is removed from the method of writing, I get the
>> following exceptions when reading. Before the removal, no such
>> exceptions.
>>
>> Could you help me how to solve it?
>>
>> Thanks so much!
>>
>> Best wishes,
>> Bing
>>
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
>> [java] WARNING: Unexpected exception receiving call responses
>> [java] java.lang.NullPointerException
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.client.ScannerCallable close
>> [java] WARNING: Ignore, probably already closed
>> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
>> failed on local exception: java.io.IOException: Unexpected exception
>> receiving call responses
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
>> [java] at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
>> [java] at $Proxy6.close(Unknown Source)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>
Sent: Wednesday, February 6, 2013 2:31 AM
Subject: Re: Is "synchronized" required?
Dear Lars,
I am now running HBase in the pseudo-distributed mode. The updated
HTable constructor also works?
Thanks so much!
Bing
On Wed, Feb 6, 2013 at 3:44 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> Don't use a pool at all.
> With HBASE-4805 (https://issues.apache.org/jira/browse/HBASE-4805) you can precreate an HConnection and ExecutorService and then make HTable cheaply on demand every time you need one.
>
> Checkout HConnectionManager.createConnection(...) and the HTable constructors.
>
> I need to document this somewhere.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[EMAIL PROTECTED]>
> To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>; "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Tuesday, February 5, 2013 10:36 PM
> Subject: Re: Is "synchronized" required?
>
> Lars,
>
> I found that at least the exceptions have nothing to do with shared HTable.
>
> To save the resources, I designed a pool for the classes that write
> and read from HBase. The primary resources consumed in the classes are
> HTable. The pool has some bugs.
>
> My question is whether it is necessary to design such a pool? Is it
> fine to create a instance of HTable for each thread?
>
> I noticed that HBase has a class, HTablePool. Maybe the pool I
> designed is NOT required?
>
> Thanks so much!
>
> Best wishes!
> Bing
>
> On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>> Are you sharing this.rankTable between threads? HTable is not thread safe.
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Bing Li <[EMAIL PROTECTED]>
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; user <[EMAIL PROTECTED]>
>> Sent: Tuesday, February 5, 2013 8:54 AM
>> Subject: Re: Is "synchronized" required?
>>
>> Dear all,
>>
>> After "synchronized" is removed from the method of writing, I get the
>> following exceptions when reading. Before the removal, no such
>> exceptions.
>>
>> Could you help me how to solve it?
>>
>> Thanks so much!
>>
>> Best wishes,
>> Bing
>>
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
>> [java] WARNING: Unexpected exception receiving call responses
>> [java] java.lang.NullPointerException
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
>> [java] at
>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>> [java] Feb 6, 2013 12:21:31 AM
>> org.apache.hadoop.hbase.client.ScannerCallable close
>> [java] WARNING: Ignore, probably already closed
>> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
>> failed on local exception: java.io.IOException: Unexpected exception
>> receiving call responses
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
>> [java] at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
>> [java] at
>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
>> [java] at $Proxy6.close(Unknown Source)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
>> [java] at
>> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
>> [java] at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
Dear Lars,
Some exceptions are raised when I concurrently read data from HBase.
Each thread to read is assigned a HTable instance. The version of
HBase I used is 0.92.0.
I cannot fix the problem. Could you please help me?
Thanks so much!
Best wishes,
Bing
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
WARNING: Unexpected exception receiving call responses
java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
WARNING: Ignore, probably already closed
java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy6.close(Unknown Source)
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
at com.greatfree.hbase.rank.NodeRankRetriever.loadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
at com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
I read data from HBase concurrently with the following code.
...
ExecutorService threadPool = Executors.newFixedThreadPool(100);
LoadNodeGroupNodeRankRowKeyThread thread;
for (String nodeKey : nodeKeys)
{
// Threads are initialized and executed here.
thread = new LoadNodeGroupNodeRankRowKeyThread(nodeKey);
threadPool.execute(thread);
}
Scanner in = new Scanner(System.in);
in.nextLine();
threadPool.shutdownNow();
...
The code of LoadNodeGroupNodeRankRowKeyThread is as follows,
...
public void run()
{
NodeRankRetriever retriever = new NodeRankRetriever();
// The following line reader data from HBase.
Set<String> rowKeys retriever.loadNodeGroupNodeRankRowKeys(this.hostNodeKey);
if (rowKeys.size() > 0)
{
for (String rowKey : rowKeys)
{
System.out.println(rowKey);
}
}
else
{
System.out.println("No data loaded");
}
retriever.dispose();
}
...
The constructor of NodeRankRetriever() just got an instance of HTable
from HTablePool from the following method.
...
public HTableInterface getTable(String tableName)
{
return this.hTablePool.getTable(tableName);
}
...
The method dispose() of NodeRankRetriever() just close the
HTableInterface created by HTablePool.
...
public void dispose()
{
try
{
this.rankTable.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
...
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
Some exceptions are raised when I concurrently read data from HBase.
Each thread to read is assigned a HTable instance. The version of
HBase I used is 0.92.0.
I cannot fix the problem. Could you please help me?
Thanks so much!
Best wishes,
Bing
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
WARNING: Unexpected exception receiving call responses
java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
WARNING: Ignore, probably already closed
java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy6.close(Unknown Source)
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
at com.greatfree.hbase.rank.NodeRankRetriever.loadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
at com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
I read data from HBase concurrently with the following code.
...
ExecutorService threadPool = Executors.newFixedThreadPool(100);
LoadNodeGroupNodeRankRowKeyThread thread;
for (String nodeKey : nodeKeys)
{
// Threads are initialized and executed here.
thread = new LoadNodeGroupNodeRankRowKeyThread(nodeKey);
threadPool.execute(thread);
}
Scanner in = new Scanner(System.in);
in.nextLine();
threadPool.shutdownNow();
...
The code of LoadNodeGroupNodeRankRowKeyThread is as follows,
...
public void run()
{
NodeRankRetriever retriever = new NodeRankRetriever();
// The following line reader data from HBase.
Set<String> rowKeys retriever.loadNodeGroupNodeRankRowKeys(this.hostNodeKey);
if (rowKeys.size() > 0)
{
for (String rowKey : rowKeys)
{
System.out.println(rowKey);
}
}
else
{
System.out.println("No data loaded");
}
retriever.dispose();
}
...
The constructor of NodeRankRetriever() just got an instance of HTable
from HTablePool from the following method.
...
public HTableInterface getTable(String tableName)
{
return this.hTablePool.getTable(tableName);
}
...
The method dispose() of NodeRankRetriever() just close the
HTableInterface created by HTablePool.
...
public void dispose()
{
try
{
this.rankTable.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
...
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
Maybe you ran into HBASE-6651 (https://issues.apache.org/jira/browse/HBASE-6651)
HTablePool is bad and we should just deprecate and remove it.
Lemme run a vote on the dev list.
-- Lars
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>
Sent: Thursday, February 7, 2013 12:10 AM
Subject: Re: Is "synchronized" required?
Dear Lars,
Some exceptions are raised when I concurrently read data from HBase.
Each thread to read is assigned a HTable instance. The version of
HBase I used is 0.92.0.
I cannot fix the problem. Could you please help me?
Thanks so much!
Best wishes,
Bing
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
WARNING: Unexpected exception receiving call responses
java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
WARNING: Ignore, probably already closed
java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy6.close(Unknown Source)
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
at com.greatfree.hbase.rank.NodeRankRetriever.loadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
at com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
I read data from HBase concurrently with the following code.
...
ExecutorService threadPool = Executors.newFixedThreadPool(100);
LoadNodeGroupNodeRankRowKeyThread thread;
for (String nodeKey : nodeKeys)
{
// Threads are initialized and executed here.
thread = new LoadNodeGroupNodeRankRowKeyThread(nodeKey);
threadPool.execute(thread);
}
Scanner in = new Scanner(System.in);
in.nextLine();
threadPool.shutdownNow();
...
The code of LoadNodeGroupNodeRankRowKeyThread is as follows,
...
public void run()
{
NodeRankRetriever retriever = new NodeRankRetriever();
// The following line reader data from HBase.
Set<String> rowKeys retriever.loadNodeGroupNodeRankRowKeys(this.hostNodeKey);
if (rowKeys.size() > 0)
{
for (String rowKey : rowKeys)
{
System.out.println(rowKey);
}
}
else
{
System.out.println("No data loaded");
}
retriever.dispose();
}
...
The constructor of NodeRankRetriever() just got an instance of HTable
from HTablePool from the following method.
...
public HTableInterface getTable(String tableName)
{
return this.hTablePool.getTable(tableName);
}
...
The method dispose() of NodeRankRetriever() just close the
HTableInterface created by HTablePool.
...
public void dispose()
{
try
{
this.rankTable.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
...
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
HTablePool is bad and we should just deprecate and remove it.
Lemme run a vote on the dev list.
-- Lars
________________________________
From: Bing Li <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>; lars hofhansl <[EMAIL PROTECTED]>
Sent: Thursday, February 7, 2013 12:10 AM
Subject: Re: Is "synchronized" required?
Dear Lars,
Some exceptions are raised when I concurrently read data from HBase.
Each thread to read is assigned a HTable instance. The version of
HBase I used is 0.92.0.
I cannot fix the problem. Could you please help me?
Thanks so much!
Best wishes,
Bing
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
WARNING: Unexpected exception receiving call responses
java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
WARNING: Ignore, probably already closed
java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy6.close(Unknown Source)
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
at com.greatfree.hbase.rank.NodeRankRetriever.loadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
at com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.io.IOException: Unexpected exception receiving call responses
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
I read data from HBase concurrently with the following code.
...
ExecutorService threadPool = Executors.newFixedThreadPool(100);
LoadNodeGroupNodeRankRowKeyThread thread;
for (String nodeKey : nodeKeys)
{
// Threads are initialized and executed here.
thread = new LoadNodeGroupNodeRankRowKeyThread(nodeKey);
threadPool.execute(thread);
}
Scanner in = new Scanner(System.in);
in.nextLine();
threadPool.shutdownNow();
...
The code of LoadNodeGroupNodeRankRowKeyThread is as follows,
...
public void run()
{
NodeRankRetriever retriever = new NodeRankRetriever();
// The following line reader data from HBase.
Set<String> rowKeys retriever.loadNodeGroupNodeRankRowKeys(this.hostNodeKey);
if (rowKeys.size() > 0)
{
for (String rowKey : rowKeys)
{
System.out.println(rowKey);
}
}
else
{
System.out.println("No data loaded");
}
retriever.dispose();
}
...
The constructor of NodeRankRetriever() just got an instance of HTable
from HTablePool from the following method.
...
public HTableInterface getTable(String tableName)
{
return this.hTablePool.getTable(tableName);
}
...
The method dispose() of NodeRankRetriever() just close the
HTableInterface created by HTablePool.
...
public void dispose()
{
try
{
this.rankTable.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
...
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[EMAIL PROTECTED]> wrote: