Re: buildbot server down?

[Kai Torben Ohlhus]

On 9/6/20 12:32 AM, John W. Eaton wrote:
> On 9/5/20 10:17 AM, John W. Eaton wrote:
>> On 9/4/20 9:59 PM, John W. Eaton wrote:
>>
>> Thinking that maybe we were just running near the edge on resource
>> limits, I resized the droplet from 2GB to 4GB RAM.  When buildbot
>> starts, the python process uses about 80MB.  But it seems to grow
>> fairly rapidly.  Here is the output from top after a few hours:
>>
>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
>> COMMAND
>>    931 buildbot  20   0 3141096   3.0g   9780 R 103.0  76.4 111:40.02
>> python3
>>
>> So it seems something is wrong there.  I don't think the buildbot
>> server process should need that much memory and I don't recall this
>> being a problem previously.
>>
>> I updated to a newer version.   We were using 2.7.0-1, now we have
>> 2.8.2-3.  I'll watch and see if that improves things.
> 
> Hmm, seems to be doing the same thing again after 90 minutes or so we
> are up to
> 
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
>  1148 buildbot  20   0 3142408   3.0g   9636 S  75.0  76.5  56:44.73
> python3
> 
> I have no idea why this is happening.  Is it buildbot?  Python 3?
> Something wrong with our configuration?
> 
> jwe
> 

Do I get the setup right, that only the Buildbot Master runs in that
Digital Ocean droplet?  Or anything else?

To my experience, there is no need for the Buildbot Master to use up
that much CPU and RAM.  My instance is not as busy as yours, but all
files (sqlite database, log files, etc.) are below 1 MB and the CPU is
mostly sleeping, RAM consumption is minimal.

In another thread [1], a tool called buildbot-profiler [2] is used.
Maybe this can be of some help to find the problem?

Kai


[1] https://github.com/buildbot/buildbot/issues/3444
[2] https://pypi.python.org/pypi/buildbot-profiler/