The application has shut down by itself

Hello,
Recently, I have problems with the application on the server.

i had 3 times situation that datagerry consumed all memory resources on the server
i have 8gb RAM and 8 gb swap on server and all was consumed by datagerry then the app started to freeze and i had to restart / kill processes on it.

i saw these information logs:
[WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)

but worse was today
2.
application was started o last Friday.

information

Aug 20 09:26:31 server datagerry[1206082]: #033[32m
Aug 20 09:26:31 server datagerry[1206082]: ########################################################################
Aug 20 09:26:31 server datagerry[1206082]:
Aug 20 09:26:31 server datagerry[1206082]: @@@@@ @ @@@@@@@ @ @@@@@ @@@@@@@ @@@@@ @@@@@ @@ @@
Aug 20 09:26:31 server datagerry[1206082]: @ @ @@ @ @@ @@@@@@@ @@@@@@@ @@@@@@ @@@@@@ @@@ @@@
Aug 20 09:26:31 server datagerry[1206082]: @ @ @ @ @ @ @ @@@ @@ @@@ @@ @@ @@ @@ @@ @@
Aug 20 09:26:31 server datagerry[1206082]: @ @ @ @ @ @ @ @@ @@@@@@ @@ @@ @@ @@ @@@@
Aug 20 09:26:31 server datagerry[1206082]: @ @ @ @ @ @ @ @@ @@@ @@@@@@ @@@@@@ @@@@@@ @@@@
Aug 20 09:26:31 server datagerry[1206082]: @ @ @@@@@@ @ @@@@@@ @@ @@@ @@@ @@@@@ @@@@@ @@
Aug 20 09:26:31 server datagerry[1206082]: @ @ @ @ @ @ @ @@@ @@ @@@ @@ @@@ @@ @@@ @@
Aug 20 09:26:31 server datagerry[1206082]: @ @ @ @ @ @ @ @@@@@@@ @@@@@@@ @@ @@@ @@ @@@ @@
Aug 20 09:26:31 server datagerry[1206082]: @@@@@ @ @ @ @ @ @@@@@@ @@@@@@@ @@ @@@ @@ @@@ @@
Aug 20 09:26:31 server datagerry[1206082]:
Aug 20 09:26:31 server datagerry[1206082]: ########################################################################
Aug 20 09:26:31 server datagerry[1206082]: #033[0m
Aug 20 09:26:31 server datagerry[1206082]: #033[33mWelcome to DATAGERRY
Aug 20 09:26:31 server datagerry[1206082]: Starting system with following parameters:
Aug 20 09:26:31 server datagerry[1206082]: {‘keys’: False, ‘debug’: False, ‘start’: True, ‘config_file’: ‘/etc/datagerry/cmdb.conf’}
Aug 20 09:26:31 server datagerry[1206082]: #033[0m
Aug 20 09:26:31 server datagerry[1206082]: #033[33mCopyright (C) 2019 NETHINKS GmbH
Aug 20 09:26:31 server datagerry[1206082]: licensed under the terms of the GNU Affero General Public License version 3
Aug 20 09:26:31 server datagerry[1206082]: #033[0m
Aug 20 09:26:31 server datagerry[1206082]: [2021-08-20 09:26:31][INFO ] — start exportd … (service.py)
Aug 20 09:26:31 server datagerry[1206082]: [2021-08-20 09:26:31][INFO ] — exportd: start run (service.py)
Aug 20 09:26:31 server datagerry[1206082]: [2021-08-20 09:26:31][INFO ] — Process manager started: True (main.py)
Aug 20 09:26:31 server datagerry[1206082]: [2021-08-20 09:26:31][INFO ] — start webapp … (service.py)
Aug 20 09:26:31 server datagerry[1206082]: [2021-08-20 09:26:31][INFO ] — DATAGERRY successfully started (main.py)
Aug 20 09:26:32 server datagerry[1206082]: [2021-08-20 09:26:32][INFO ] — Interfaces started @ http://0.0.0.0:4000 (gunicorn.py)

When i tried today logged to datagerry i got nothing application don’t work.
No datagerry process was running. the application home directory at / tmp / did not exist. The application simply folded

Service information

[root@server ~]# systemctl status datagerry
● datagerry.service - DATAGERRY - Enterprise grade OpenSource CMDB
Loaded: loaded (/etc/systemd/system/datagerry.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Sat 2021-08-21 16:23:49 CEST; 1 day 13h ago
Process: 1206082 ExecStart=/usr/bin/datagerry -c /etc/datagerry/cmdb.conf -s (code=exited, status=0/SUCCESS)
Main PID: 1206082 (code=exited, status=0/SUCCESS)

Aug 21 16:23:27 server datagerry[1206082]: [2021-08-21 16:23:27][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 16:23:27 server datagerry[1206082]: [2021-08-21 16:23:27][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][ERROR ] — exportd: EventSenderAmqp connection error (event_manager.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][ERROR ] — exportd: EventReceiverAmqp connection error (event_manager.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][INFO ] — shutdown exportd … (service.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][INFO ] — exportd: end run (service.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][INFO ] — shutdown exportd completed (service.py)
Aug 21 16:23:44 server datagerry[1206082]: [2021-08-21 16:23:44][INFO ] — shutdown webapp … (service.py)
Aug 21 16:23:46 server datagerry[1206082]: [2021-08-21 16:23:46][INFO ] — shutdown webapp completed (service.py)
Aug 21 16:23:49 server systemd[1]: datagerry.service: Succeeded.

in /var/log/messages i got only this

logs

Aug 20 13:09:22 server datagerry[1206082]: [2021-08-20 13:09:22][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 13:09:22 server datagerry[1206082]: [2021-08-20 13:09:22][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 13:42:38 server datagerry[1206082]: [2021-08-20 13:42:38][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 13:42:38 server datagerry[1206082]: [2021-08-20 13:42:38][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 14:32:56 server datagerry[1206082]: [2021-08-20 14:32:56][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 14:32:56 server datagerry[1206082]: [2021-08-20 14:32:56][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 16:10:57 server datagerry[1206082]: [2021-08-20 16:10:57][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 16:10:57 server datagerry[1206082]: [2021-08-20 16:10:57][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 16:23:40 server datagerry[1206082]: [2021-08-20 16:23:40][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 18:49:24 server datagerry[1206082]: [2021-08-20 18:49:23][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 18:49:24 server datagerry[1206082]: [2021-08-20 18:49:24][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 18:59:34 server datagerry[1206082]: [2021-08-20 18:59:34][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 18:59:34 server datagerry[1206082]: [2021-08-20 18:59:34][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 23:25:49 server datagerry[1206082]: [2021-08-20 23:25:48][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 23:25:49 server datagerry[1206082]: [2021-08-20 23:25:48][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 23:36:49 server datagerry[1206082]: [2021-08-20 23:36:49][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 20 23:36:49 server datagerry[1206082]: [2021-08-20 23:36:49][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 04:56:01 server datagerry[1206082]: [2021-08-21 04:56:00][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 04:56:01 server datagerry[1206082]: [2021-08-21 04:56:01][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 10:01:11 server datagerry[1206082]: [2021-08-21 10:01:11][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 10:01:12 server datagerry[1206082]: [2021-08-21 10:01:11][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 10:30:27 server datagerry[1206082]: [2021-08-21 10:30:27][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 10:30:27 server datagerry[1206082]: [2021-08-21 10:30:27][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 15:58:37 server datagerry[1206082]: [2021-08-21 15:58:35][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 15:58:37 server datagerry[1206082]: [2021-08-21 15:58:37][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 16:23:27 server datagerry[1206082]: [2021-08-21 16:23:27][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 16:23:27 server datagerry[1206082]: [2021-08-21 16:23:27][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][ERROR ] — exportd: EventSenderAmqp connection error (event_manager.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][ERROR ] — exportd: EventReceiverAmqp connection error (event_manager.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][INFO ] — shutdown exportd … (service.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][INFO ] — exportd: end run (service.py)
Aug 21 16:23:42 server datagerry[1206082]: [2021-08-21 16:23:42][INFO ] — shutdown exportd completed (service.py)
Aug 21 16:23:44 server datagerry[1206082]: [2021-08-21 16:23:44][INFO ] — shutdown webapp … (service.py)
Aug 21 16:23:46 server datagerry[1206082]: [2021-08-21 16:23:46][INFO ] — shutdown webapp completed (service.py)
Aug 21 16:23:49 server systemd[1]: datagerry.service: Succeeded.

Please check it before version 1.6.4 work everything and app don’t had problem with connections

System:
CentOS Stream release 8

[root@server~]# rpm -qa |grep rabbit
rabbitmq-server-3.8.14-1.el8.noarch
[root@server~]# rpm -qa |grep mongo
mongodb-org-shell-4.2.15-1.el8.x86_64
mongodb-org-mongos-4.2.15-1.el8.x86_64
mongodb-org-tools-4.2.15-1.el8.x86_64
mongodb-org-4.2.15-1.el8.x86_64
mongodb-org-server-4.2.15-1.el8.x86_64

Today again:
Aug 25 13:40:05 server datagerry[1552653]: [2021-08-25 13:40:04][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 25 13:40:06 server datagerry[1552653]: [2021-08-25 13:40:02][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
Aug 25 13:41:05 server datagerry[1552653]: [2021-08-25 13:41:05][ERROR ] — exportd: EventSenderAmqp connection error (event_manager.py)
Aug 25 13:41:06 server datagerry[1552653]: [2021-08-25 13:41:06][INFO ] — shutdown exportd … (service.py)
Aug 25 13:41:06 server datagerry[1552653]: [2021-08-25 13:41:06][INFO ] — exportd: end run (service.py)
Aug 25 13:41:07 server datagerry[1552653]: [2021-08-25 13:41:07][ERROR ] — exportd: EventReceiverAmqp connection error (event_manager.py)
Aug 25 13:41:07 server datagerry[1552653]: [2021-08-25 13:41:07][INFO ] — shutdown exportd completed (service.py)
Aug 25 13:41:09 server datagerry[1552653]: [2021-08-25 13:41:09][INFO ] — shutdown webapp … (service.py)
Aug 25 13:41:13 server datagerry[1552653]: [2021-08-25 13:41:13][INFO ] — shutdown webapp completed (service.py)

app crash

[root@server ~]$ ps -aux | grep datagerry
root 1840601 0.0 0.0 221928 1160 pts/3 S+ 13:44 0:00 grep --color=auto datagerry

Hi @marcinw,

we will look at the behavior. as soon as we know more. we will write to you.

With kind regards

Hello,
today i have situation:

in logs:

[2021-09-13 17:17:35][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-13 17:17:35][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-13 17:33:09][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 08:12:11][ERROR ] — E11000 duplicate key error collection: cmdb.exportd.logs index: public_id dup key: { public_id: 25821 }, full error: {
‘index’: 0, ‘code’: 11000, ‘keyPattern’: {‘public_id’: 1}, ‘keyValue’: {‘public_id’: 25821}, ‘errmsg’: ‘E11000 duplicate key error collection: cmdb.exportd.logs index: public_id dup key: { public_id: 25821 }’} (exporter_base.py)
[2021-09-15 10:33:41][ERROR ] — E11000 duplicate key error collection: cmdb.exportd.logs index: public_id dup key: { public_id: 25897 }, full error: {‘index’: 0, ‘code’: 11000, ‘keyPattern’: {‘public_id’: 1}, ‘keyValue’: {‘public_id’: 25897}, ‘errmsg’: ‘E11000 duplicate key error collection: cmdb.exportd.logs index: public_id dup key: { public_id: 25897 }’} (exporter_base.py)
[2021-09-15 12:07:11][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 12:07:11][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 12:20:30][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 12:20:31][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 14:13:44][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 14:13:45][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 14:38:31][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 14:38:31][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 16:00:56][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 16:00:56][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 16:07:12][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 16:07:12][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-15 16:18:03][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 00:09:29][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 00:09:29][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 00:28:58][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 00:28:58][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 02:19:03][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 02:19:03][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 04:23:19][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 04:23:19][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 04:34:52][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 04:34:52][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 04:39:42][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 04:39:42][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)
[2021-09-16 04:42:54][WARNING ] — connection to broker lost, try to reconnect… (event_manager.py)

and datagarry on server has consumed
free -m
total used free shared buff/cache available
Mem: 7768 7494 127 1 146 66
Swap: 8103 4051 4052

top

top - 06:57:30 up 21 days, 17:04, 2 users, load average: 1,89, 2,01, 1,91
Tasks: 238 total, 2 running, 236 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2,6 us, 2,3 sy, 0,0 ni, 71,5 id, 22,8 wa, 0,7 hi, 0,2 si, 0,0 st
MiB Mem : 7768,1 total, 130,3 free, 7513,4 used, 124,4 buff/cache
MiB Swap: 8104,0 total, 4052,2 free, 4051,8 used. 58,6 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

1513793 datager+ 20 0 10,3g 6,0g 2220 S 8,6 79,0 1375:33 datagerry
1513803 datager+ 20 0 864552 62944 3268 S 1,0 0,8 77:43.70 datagerry
1513811 datager+ 20 0 832736 60924 3064 S 1,0 0,8 77:49.16 datagerry
1513805 datager+ 20 0 877588 60176 3008 S 0,7 0,8 78:17.53 datagerry
1513807 datager+ 20 0 873156 58192 3088 S 0,7 0,7 78:07.33 datagerry
1513813 datager+ 20 0 854420 65768 3044 S 0,7 0,8 77:47.41 datagerry
1513815 datager+ 20 0 865792 78220 3148 S 0,7 1,0 77:30.90 datagerry
56 root 0 -20 0 0 0 I 0,3 0,0 0:58.66 kworker/0:1H-kblockd
346 root 0 -20 0 0 0 I 0,3 0,0 1:04.10 kworker/2:1H-kblockd
1513809 datager+ 20 0 850608 61104 3252 S 0,3 0,8 78:04.14 datagerry
1513817 datager+ 20 0 862128 53828 3140 S 0,3 0,7 77:54.50 datagerry
1513819 datager+ 20 0 882912 60388 3108 S 0,3 0,8 78:16.51 datagerry
2568803 user 20 0 126668 4456 3100 S 0,3 0,1 0:00.28 sshd
2571785 root 20 0 266312 4764 3808 R 0,3 0,1 0:00.02 top

Hello,
in logs rabbitmq i got errors:

example part of log

2021-09-16 04:42:42.260 [error] <0.1964.14> closing AMQP connection <0.1964.14> (127.0.0.1:43662 → 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s
2021-09-16 04:42:42.260 [info] <0.2039.14> Closing all channels from connection ‘127.0.0.1:43662 → 127.0.0.1:5672’ because it has been closed
2021-09-16 04:42:42.403 [error] <0.1961.14> closing AMQP connection <0.1961.14> (127.0.0.1:43660 → 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s
2021-09-16 04:42:42.403 [info] <0.2044.14> Closing all channels from connection ‘127.0.0.1:43660 → 127.0.0.1:5672’ because it has been closed
2021-09-16 04:42:54.439 [info] <0.2058.14> accepting AMQP connection <0.2058.14> (127.0.0.1:43676 → 127.0.0.1:5672)
2021-09-16 04:42:54.440 [info] <0.2055.14> accepting AMQP connection <0.2055.14> (127.0.0.1:43672 → 127.0.0.1:5672)
2021-09-16 04:42:54.496 [info] <0.2055.14> connection <0.2055.14> (127.0.0.1:43672 → 127.0.0.1:5672): user ‘guest’ authenticated and granted access to vhost ‘/’
2021-09-16 04:42:54.497 [info] <0.2058.14> connection <0.2058.14> (127.0.0.1:43676 → 127.0.0.1:5672): user ‘guest’ authenticated and granted access to vhost ‘/’
2021-09-16 04:48:22.947 [warning] <0.2058.14> closing AMQP connection <0.2058.14> (127.0.0.1:43676 → 127.0.0.1:5672, vhost: ‘/’, user: ‘guest’):
client unexpectedly closed TCP connection
2021-09-16 04:48:22.948 [info] <0.2176.14> Closing all channels from connection ‘127.0.0.1:43676 → 127.0.0.1:5672’ because it has been closed
2021-09-16 04:48:22.992 [info] <0.2183.14> accepting AMQP connection <0.2183.14> (127.0.0.1:43690 → 127.0.0.1:5672)
2021-09-16 04:48:23.147 [info] <0.2183.14> connection <0.2183.14> (127.0.0.1:43690 → 127.0.0.1:5672): user ‘guest’ authenticated and granted access to vhost ‘/’
2021-09-16 04:50:54.504 [error] <0.2055.14> closing AMQP connection <0.2055.14> (127.0.0.1:43672 → 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s