Hello,
How to explain that:

2011-08-08T19:02:24.947+0000: 14564.360: [GC 14564.360: [ParNew:
52681K->6528K(59008K), 158.6620830 secs] 2285054K->2240073K(4187776K)
icms_dc=0 , 158.6622690 secs] [Times: user=0.00 sys=0.00, real=158.65 secs]
Total time for which application threads were stopped: 158.6632510 seconds
QUESTION: How can? Zero, Zero, and real=158 seconds?!!
However, in most cases (99.99%) I have
2011-08-08T18:55:20.811+0000: 14140.225: [GC 14140.225: [ParNew:
54381K->2110K(59008K), 0.0286450 secs] 1651815K->1599544K(4187776K)
icms_dc=0 , 0.0287720 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0296330 seconds
2011-08-08T18:55:21.181+0000: 14140.594: [GC 14140.594: [ParNew:
54588K->3056K(59008K), 0.0388800 secs] 1652022K->1600490K(4187776K)
icms_dc=0 , 0.0390230 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
Total time for which application threads were stopped: 0.0398510 seconds
2011-08-08T18:55:21.527+0000: 14140.940: [GC 14140.940: [ParNew:
55536K->4812K(59008K), 0.0326740 secs] 1652970K->1602247K(4187776K)
icms_dc=0 , 0.0328500 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
Total time for which application threads were stopped: 0.0338200 seconds
2011-08-08T18:55:21.868+0000: 14141.281: [GC 14141.281: [ParNew:
57258K->5232K(59008K), 0.0091650 secs] 1654693K->1604268K(4187776K)
icms_dc=0 , 0.0093210 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
Total time for which application threads were stopped: 0.0101470 seconds
2011-08-08T18:55:22.179+0000: 14141.592: [GC 14141.592: [ParNew:
57712K->6521K(59008K), 0.0307120 secs] 1656748K->1608576K(4187776K)
icms_dc=0 , 0.0308700 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
Total time for which application threads were stopped: 0.0318530 seconds
2011-08-08T18:55:22.603+0000: 14142.016: [GC 14142.016: [ParNew:
59001K->364K(59008K), 0.0471390 secs] 1661056K->1608313K(4187776K) icms_dc=0
, 0.0472720 secs] [Times: user=0.00 sys=0.00, real=0.05 secs]
Total time for which application threads were stopped: 0.0480990 seconds
2011-08-08T18:55:22.994+0000: 14142.407: [GC 14142.407: [ParNew:
52844K->2014K(59008K), 0.0257020 secs] 1660793K->1609963K(4187776K)
icms_dc=0 , 0.0258530 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
I believe we are indeed using "virtualization"Š $0.68/hour, three nodes
7 GB of memory
20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: c1.xlarge
I am absolutely sure this is unpredictable and cause of a problem is sharing
the same hardware with 3-4 other users. Box such as cc1.4xlarge is shared
with 3 other users to make 3 instances of c1.xlarge, and no swap file (but
"swap caching"?)
23 GB of memory
33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core ³Nehalem²
architecture)
1690 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
API name: cc1.4xlarge

Any EC2 bad/good experience to share?

Thanks,
--
Fuad Efendi
http://www.tokenizer.ca

Looks normal to me considering the platform. It's not so much a GC as
the machine was unavailable for more than 2 minutes since there's no
user or sys CPU involved.

J-D

On Mon, Aug 8, 2011 at 12:27 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
> Hello,
>
>
> How to explain that:
>
> 2011-08-08T19:02:24.947+0000: 14564.360: [GC 14564.360: [ParNew:
> 52681K->6528K(59008K), 158.6620830 secs] 2285054K->2240073K(4187776K)
> icms_dc=0 , 158.6622690 secs] [Times: user=0.00 sys=0.00, real=158.65 secs]
> Total time for which application threads were stopped: 158.6632510 seconds
>
>
> QUESTION: How can? Zero, Zero, and real=158 seconds?!!
>
>
> However, in most cases (99.99%) I have
> 2011-08-08T18:55:20.811+0000: 14140.225: [GC 14140.225: [ParNew:
> 54381K->2110K(59008K), 0.0286450 secs] 1651815K->1599544K(4187776K)
> icms_dc=0 , 0.0287720 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.0296330 seconds
> 2011-08-08T18:55:21.181+0000: 14140.594: [GC 14140.594: [ParNew:
> 54588K->3056K(59008K), 0.0388800 secs] 1652022K->1600490K(4187776K)
> icms_dc=0 , 0.0390230 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
> Total time for which application threads were stopped: 0.0398510 seconds
> 2011-08-08T18:55:21.527+0000: 14140.940: [GC 14140.940: [ParNew:
> 55536K->4812K(59008K), 0.0326740 secs] 1652970K->1602247K(4187776K)
> icms_dc=0 , 0.0328500 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
> Total time for which application threads were stopped: 0.0338200 seconds
> 2011-08-08T18:55:21.868+0000: 14141.281: [GC 14141.281: [ParNew:
> 57258K->5232K(59008K), 0.0091650 secs] 1654693K->1604268K(4187776K)
> icms_dc=0 , 0.0093210 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
> Total time for which application threads were stopped: 0.0101470 seconds
> 2011-08-08T18:55:22.179+0000: 14141.592: [GC 14141.592: [ParNew:
> 57712K->6521K(59008K), 0.0307120 secs] 1656748K->1608576K(4187776K)
> icms_dc=0 , 0.0308700 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
> Total time for which application threads were stopped: 0.0318530 seconds
> 2011-08-08T18:55:22.603+0000: 14142.016: [GC 14142.016: [ParNew:
> 59001K->364K(59008K), 0.0471390 secs] 1661056K->1608313K(4187776K) icms_dc=0
> , 0.0472720 secs] [Times: user=0.00 sys=0.00, real=0.05 secs]
> Total time for which application threads were stopped: 0.0480990 seconds
> 2011-08-08T18:55:22.994+0000: 14142.407: [GC 14142.407: [ParNew:
> 52844K->2014K(59008K), 0.0257020 secs] 1660793K->1609963K(4187776K)
> icms_dc=0 , 0.0258530 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
>
>
> I believe we are indeed using "virtualization"Š $0.68/hour, three nodes
> 7 GB of memory
> 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
> 1690 GB of instance storage
> 64-bit platform
> I/O Performance: High
> API name: c1.xlarge
>
>
> I am absolutely sure this is unpredictable and cause of a problem is sharing
> the same hardware with 3-4 other users. Box such as cc1.4xlarge is shared
> with 3 other users to make 3 instances of c1.xlarge, and no swap file (but
> "swap caching"?)
> 23 GB of memory
> 33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core ³Nehalem��
> architecture)
> 1690 GB of instance storage
> 64-bit platform
> I/O Performance: Very High (10 Gigabit Ethernet)
> API name: cc1.4xlarge
>
>
>
> Any EC2 bad/good experience to share?
>
>
>
>
>
> Thanks,
>
>
> --
> Fuad Efendi
> http://www.tokenizer.ca
>
>
>
>
>
>

Then how to explain this unpredictable 629 seconds with completely
underloaded server:
Total time for which application threads were stopped: 0.0004060 seconds
Total time for which application threads were stopped: 629.5515080 seconds
(!)
Total time for which application threads were stopped: 0.0006960 seconds
Total time for which application threads were stopped: 0.0001590 seconds
Total time for which application threads were stopped: 0.0003250 seconds

And this is still unclear too:
[Times: user=0.00 sys=0.00, real=158.65 secs]
What is "real" in EC2 terms? Broken router?!


On 11-08-29 2:39 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:

>Looks normal to me considering the platform. It's not so much a GC as
>the machine was unavailable for more than 2 minutes since there's no
>user or sys CPU involved.
>
>J-D
>
>On Mon, Aug 8, 2011 at 12:27 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>>
>> How to explain that:
>>
>> 2011-08-08T19:02:24.947+0000: 14564.360: [GC 14564.360: [ParNew:
>> 52681K->6528K(59008K), 158.6620830 secs] 2285054K->2240073K(4187776K)
>> icms_dc=0 , 158.6622690 secs] [Times: user=0.00 sys=0.00, real=158.65
>>secs]
>> Total time for which application threads were stopped: 158.6632510
>>seconds
>>
>>
>> QUESTION: How can? Zero, Zero, and real=158 seconds?!!
>>
>>
>> However, in most cases (99.99%) I have
>> 2011-08-08T18:55:20.811+0000: 14140.225: [GC 14140.225: [ParNew:
>> 54381K->2110K(59008K), 0.0286450 secs] 1651815K->1599544K(4187776K)
>> icms_dc=0 , 0.0287720 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
>> Total time for which application threads were stopped: 0.0296330 seconds
>> 2011-08-08T18:55:21.181+0000: 14140.594: [GC 14140.594: [ParNew:
>> 54588K->3056K(59008K), 0.0388800 secs] 1652022K->1600490K(4187776K)
>> icms_dc=0 , 0.0390230 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
>> Total time for which application threads were stopped: 0.0398510 seconds
>> 2011-08-08T18:55:21.527+0000: 14140.940: [GC 14140.940: [ParNew:
>> 55536K->4812K(59008K), 0.0326740 secs] 1652970K->1602247K(4187776K)
>> icms_dc=0 , 0.0328500 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
>> Total time for which application threads were stopped: 0.0338200 seconds
>> 2011-08-08T18:55:21.868+0000: 14141.281: [GC 14141.281: [ParNew:
>> 57258K->5232K(59008K), 0.0091650 secs] 1654693K->1604268K(4187776K)
>> icms_dc=0 , 0.0093210 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>> Total time for which application threads were stopped: 0.0101470 seconds
>> 2011-08-08T18:55:22.179+0000: 14141.592: [GC 14141.592: [ParNew:
>> 57712K->6521K(59008K), 0.0307120 secs] 1656748K->1608576K(4187776K)
>> icms_dc=0 , 0.0308700 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
>> Total time for which application threads were stopped: 0.0318530 seconds
>> 2011-08-08T18:55:22.603+0000: 14142.016: [GC 14142.016: [ParNew:
>> 59001K->364K(59008K), 0.0471390 secs] 1661056K->1608313K(4187776K)
>>icms_dc=0
>> , 0.0472720 secs] [Times: user=0.00 sys=0.00, real=0.05 secs]
>> Total time for which application threads were stopped: 0.0480990 seconds
>> 2011-08-08T18:55:22.994+0000: 14142.407: [GC 14142.407: [ParNew:
>> 52844K->2014K(59008K), 0.0257020 secs] 1660793K->1609963K(4187776K)
>> icms_dc=0 , 0.0258530 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
>>
>>
>> I believe we are indeed using "virtualization"Š $0.68/hour, three nodes
>> 7 GB of memory
>> 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
>> 1690 GB of instance storage
>> 64-bit platform
>> I/O Performance: High
>> API name: c1.xlarge
>>
>>
>> I am absolutely sure this is unpredictable and cause of a problem is
>>sharing
>> the same hardware with 3-4 other users. Box such as cc1.4xlarge is
>>shared
>> with 3 other users to make 3 instances of c1.xlarge, and no swap file
>>(but
>> "swap caching"?)
>> 23 GB of memory
>> 33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core ³Nehalem²
>> architecture)
>> 1690 GB of instance storage

You should ask your provider.

AFAIK real time is, errr, well real time. Those machines are shared,
who knows what's going on on the other instances.

J-D

On Mon, Aug 29, 2011 at 11:48 AM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
> Then how to explain this unpredictable 629 seconds with completely
> underloaded server:
> Total time for which application threads were stopped: 0.0004060 seconds
> Total time for which application threads were stopped: 629.5515080 seconds
> (!)
> Total time for which application threads were stopped: 0.0006960 seconds
> Total time for which application threads were stopped: 0.0001590 seconds
> Total time for which application threads were stopped: 0.0003250 seconds
>
>
>
> And this is still unclear too:
> [Times: user=0.00 sys=0.00, real=158.65 secs]
>
>
> What is "real" in EC2 terms? Broken router?!
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On 11-08-29 2:39 PM, "Jean-Daniel Cryans" <[EMAIL PROTECTED]> wrote:
>
>>Looks normal to me considering the platform. It's not so much a GC as
>>the machine was unavailable for more than 2 minutes since there's no
>>user or sys CPU involved.
>>
>>J-D
>>
>>On Mon, Aug 8, 2011 at 12:27 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
>>> Hello,
>>>
>>>
>>> How to explain that:
>>>
>>> 2011-08-08T19:02:24.947+0000: 14564.360: [GC 14564.360: [ParNew:
>>> 52681K->6528K(59008K), 158.6620830 secs] 2285054K->2240073K(4187776K)
>>> icms_dc=0 , 158.6622690 secs] [Times: user=0.00 sys=0.00, real=158.65
>>>secs]
>>> Total time for which application threads were stopped: 158.6632510
>>>seconds
>>>
>>>
>>> QUESTION: How can? Zero, Zero, and real=158 seconds?!!
>>>
>>>
>>> However, in most cases (99.99%) I have
>>> 2011-08-08T18:55:20.811+0000: 14140.225: [GC 14140.225: [ParNew:
>>> 54381K->2110K(59008K), 0.0286450 secs] 1651815K->1599544K(4187776K)
>>> icms_dc=0 , 0.0287720 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
>>> Total time for which application threads were stopped: 0.0296330 seconds
>>> 2011-08-08T18:55:21.181+0000: 14140.594: [GC 14140.594: [ParNew:
>>> 54588K->3056K(59008K), 0.0388800 secs] 1652022K->1600490K(4187776K)
>>> icms_dc=0 , 0.0390230 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
>>> Total time for which application threads were stopped: 0.0398510 seconds
>>> 2011-08-08T18:55:21.527+0000: 14140.940: [GC 14140.940: [ParNew:
>>> 55536K->4812K(59008K), 0.0326740 secs] 1652970K->1602247K(4187776K)
>>> icms_dc=0 , 0.0328500 secs] [Times: user=0.00 sys=0.00, real=0.04 secs]
>>> Total time for which application threads were stopped: 0.0338200 seconds
>>> 2011-08-08T18:55:21.868+0000: 14141.281: [GC 14141.281: [ParNew:
>>> 57258K->5232K(59008K), 0.0091650 secs] 1654693K->1604268K(4187776K)
>>> icms_dc=0 , 0.0093210 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
>>> Total time for which application threads were stopped: 0.0101470 seconds
>>> 2011-08-08T18:55:22.179+0000: 14141.592: [GC 14141.592: [ParNew:
>>> 57712K->6521K(59008K), 0.0307120 secs] 1656748K->1608576K(4187776K)
>>> icms_dc=0 , 0.0308700 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
>>> Total time for which application threads were stopped: 0.0318530 seconds
>>> 2011-08-08T18:55:22.603+0000: 14142.016: [GC 14142.016: [ParNew:
>>> 59001K->364K(59008K), 0.0471390 secs] 1661056K->1608313K(4187776K)
>>>icms_dc=0
>>> , 0.0472720 secs] [Times: user=0.00 sys=0.00, real=0.05 secs]
>>> Total time for which application threads were stopped: 0.0480990 seconds
>>> 2011-08-08T18:55:22.994+0000: 14142.407: [GC 14142.407: [ParNew:
>>> 52844K->2014K(59008K), 0.0257020 secs] 1660793K->1609963K(4187776K)
>>> icms_dc=0 , 0.0258530 secs] [Times: user=0.00 sys=0.00, real=0.03 secs]
>>>
>>>
>>> I believe we are indeed using "virtualization"Š $0.68/hour, three nodes
>>> 7 GB of memory
>>> 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
>>> 1690 GB of instance storage
>>> 64-bit platform
>>> I/O Performance: High
>>> API name: c1.xlarge
>>>
>>>
>>> I am absolutely sure this is unpredictable and cause of a problem is

You don't mention the OS you're running but this is reminiscent of
Ubuntu #708920.

https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708920

This always occurred for us on the Nehalem architectures, you might
try and boot multiple instances until you get an older Xeon (these
days usually takes a dozen or so instances), see if you can repro on a
different architecture.

Yes, it is UBUNTU, but architecture is suspicious (virtualization)
Also, possible cause is EBS

Thanks for the link

On 11-08-29 2:52 PM, "Erik Onnen" <[EMAIL PROTECTED]> wrote:

>You don't mention the OS you're running but this is reminiscent of
>Ubuntu #708920.
>
>https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708920
>
>This always occurred for us on the Nehalem architectures, you might
>try and boot multiple instances until you get an older Xeon (these
>days usually takes a dozen or so instances), see if you can repro on a
>different architecture.