Datagerry borked after API call with curl to datagerry

Community Support
1

Hi,

Our working Datagerry installion just broke.

It seems to have broken after a client sent this api call:

PS Microsoft.PowerShell.Core\FileSystem::\\Client\C$\Users\xxxxx\Documents> \\Client\C$\Users\xxxxx\Documents\dg.ps1

 
results       : {@{type_id=4; status=True; version=1.0.0; creation_time=2024-05-07T13:13:45.178000; author_id=1; last_edit_time=; editor_id=; active=True; 
                fields=System.Object[]; public_id=5}, @{type_id=4; status=True; version=1.0.0; creation_time=2024-05-07T13:13:53.132000; author_id=1; last_edit_time=; 
                editor_id=; active=True; fields=System.Object[]; public_id=6}, @{type_id=4; status=True; version=1.0.0; creation_time=2024-05-07T13:13:59.312000; 
                author_id=1; last_edit_time=; editor_id=; active=True; fields=System.Object[]; public_id=7}, @{type_id=2; status=True; version=2.0.0; 
                creation_time=2024-05-16T06:47:37.860000; author_id=1; last_edit_time=2024-05-21T04:50:37.484000; editor_id=1; active=True; fields=System.Object[]; 
                public_id=8}...}
count         : 10
total         : 13
parameters    : @{limit=10; sort=public_id; order=1; page=1; filter=; optional=}
pager         : @{page=1; page_size=10; total_pages=2}
pagination    : @{current=https://dg1.example.net/rest/objects/; first=https://dg1.example.net/rest/objects/?page=1; 
                prev=https://dg1.example.net/rest/objects/?page=1; next=https://dg1.example.net/rest/objects/?page=2; 
                last=https://dg1.example.net/rest/objects/?page=2}
response_type : GET
model         : Object
time          : 2024-06-03T10:21:04.416401+00:00
 
{
    "fields":  [
                   {
                       "value":  1,
                       "name":  "vlan_id"
                   },
                   {
                       "value":  "1.1.1.1/24",
                       "name":  "cidr"
                   }
               ],
    "version":  "1.0.1",
    "type_id":  10,
    "views":  0,
    "author_id":  1,
    "active":  true
}
19

Has anybody seen any of the error messages below?

Now I get this screen when I log into the web gui ( see screenshot ).

These are the log file messages when I start this up:

Jun  3 12:16:50 dg1 datagerry[23042]: [2024-06-03 12:16:50][DEBUG   ] --- event received:cmdb.core.object.added (service.py)
Jun  3 12:16:50 dg1 datagerry[23059]: [2024-06-03 12:16:50][ERROR   ] --- string indices must be integers (objects_routes.py)
Jun  3 12:17:39 dg1 datagerry[23042]: [2024-06-03 12:17:39][DEBUG   ] --- event received:cmdb.core.object.added (service.py)
Jun  3 12:17:39 dg1 datagerry[23053]: [2024-06-03 12:17:39][ERROR   ] --- string indices must be integers (objects_routes.py)
Jun  3 12:21:04 dg1 datagerry[23042]: [2024-06-03 12:21:04][DEBUG   ] --- event received:cmdb.core.object.added (service.py)
Jun  3 14:39:09 dg1 datagerry[23034]: Exception ignored in: <module 'threading' from '/tmp/_MEIQvmNN7/threading.pyc'>
Jun  3 14:39:09 dg1 datagerry[23034]: Traceback (most recent call last):
Jun  3 14:39:09 dg1 datagerry[23034]:  File "threading.py", line 1477, in _shutdown
Jun  3 14:39:09 dg1 datagerry[23034]: TypeError: _stop_app() takes 0 positional arguments but 2 were given
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- DATAGERRY starting... (__main__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][WARNING ] --- DEBUG mode enabled (__main__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- Checking database connection with cmdb.conf data (__main__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- Database connection established. (__main__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- STARTING Checks... (__check__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- CHECK: Checking database collection validation (__check__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- CHECK: Root Location valid (__check__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- CHECK: Database collection validation status True (__check__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- FINISHED Checks! (__check__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- SETUP ROUTINE: STARTED... (__setup__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- SETUP ROUTINE: Checking database connection (__setup__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- SETUP ROUTINE: Database connection status True (__setup__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- SETUP ROUTINE: FINISHED! (__setup__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]:        ########################################################################
Jun  3 14:57:34 dg1 datagerry[1048961]:
Jun  3 14:57:34 dg1 datagerry[1048961]:        @@@@@     @   @@@@@@@ @           @@@@@  @@@@@@@ @@@@@   @@@@@  @@    @@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @    @    @@     @    @@         @@@@@@@ @@@@@@@ @@@@@@  @@@@@@ @@@  @@@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @     @  @  @    @   @  @       @@@   @@ @@@     @@   @@ @@   @@ @@  @@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @     @  @  @    @   @  @       @@       @@@@@@  @@   @@ @@   @@  @@@@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @     @ @    @   @  @    @      @@   @@@ @@@@@@  @@@@@@  @@@@@@   @@@@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @     @ @@@@@@   @  @@@@@@      @@   @@@ @@@     @@@@@   @@@@@     @@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @     @ @    @   @  @    @      @@@   @@ @@@     @@ @@@  @@ @@@    @@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @    @ @      @  @ @      @      @@@@@@@ @@@@@@@ @@  @@@ @@  @@@   @@
Jun  3 14:57:34 dg1 datagerry[1048961]:        @@@@@  @      @  @ @      @       @@@@@@ @@@@@@@ @@  @@@ @@  @@@   @@
Jun  3 14:57:34 dg1 datagerry[1048961]:
Jun  3 14:57:34 dg1 datagerry[1048961]:        ########################################################################
Jun  3 14:57:34 dg1 datagerry[1048961]:
Jun  3 14:57:34 dg1 datagerry[1048961]:        Welcome to DATAGERRY
Jun  3 14:57:34 dg1 datagerry[1048961]:        Starting system with following parameters:
Jun  3 14:57:34 dg1 datagerry[1048961]:        {'keys': False, 'debug': True, 'start': True, 'config_file': '/etc/datagerry/cmdb.conf'}
Jun  3 14:57:34 dg1 datagerry[1048961]:
Jun  3 14:57:34 dg1 datagerry[1048961]:        Copyright (C) 2024 becon GmbH
Jun  3 14:57:34 dg1 datagerry[1048961]:        licensed under the terms of the GNU Affero General Public License version 3
Jun  3 14:57:34 dg1 datagerry[1048961]:
Jun  3 14:57:34 dg1 datagerry[1048969]: [2024-06-03 14:57:34][INFO    ] --- start exportd ... (service.py)
Jun  3 14:57:34 dg1 datagerry[1048969]: [2024-06-03 14:57:34][INFO    ] --- exportd: start run (service.py)
Jun  3 14:57:34 dg1 datagerry[1048969]: Exception in thread Thread-2:
Jun  3 14:57:34 dg1 datagerry[1048969]: Traceback (most recent call last):
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "threading.py", line 980, in _bootstrap_inner
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "cmdb/event_management/event_manager.py", line 321, in run
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "cmdb/event_management/event_manager.py", line 306, in __init_connection
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/blocking_connection.py", line 2468, in queue_declare
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/blocking_connection.py", line 1290, in _flush_output
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/blocking_connection.py", line 458, in _flush_output
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/select_connection.py", line 495, in poll
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/select_connection.py", line 1114, in poll
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/select_connection.py", line 831, in _dispatch_fd_events
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/base_connection.py", line 410, in _handle_events
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/base_connection.py", line 464, in _handle_read
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/connection.py", line 2021, in _on_data_available
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/connection.py", line 2142, in _process_frame
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/connection.py", line 2120, in _process_callbacks
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/callback.py", line 60, in wrapper
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/callback.py", line 92, in wrapper
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/callback.py", line 236, in process
Jun  3 14:57:34 dg1 datagerry[1048969]:  File "pika/adapters/blocking_connection.py", line 1357, in _on_channel_closed
Jun  3 14:57:34 dg1 datagerry[1048969]: pika.exceptions.ChannelClosed: (405, "RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'datagerry.eventbus.exportd' in vhost '/'. It could be originally declared on another connection or the exclusive property value does not match that of the original declaration.")
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- Process manager started: True (__main__.py)
Jun  3 14:57:34 dg1 datagerry[1048961]: [2024-06-03 14:57:34][INFO    ] --- DATAGERRY successfully started (__main__.py)
Jun  3 14:57:34 dg1 datagerry[1048974]: [2024-06-03 14:57:34][INFO    ] --- start webapp ... (service.py)
Jun  3 14:57:34 dg1 datagerry[1048974]: Exception in thread Thread-3:
Jun  3 14:57:34 dg1 datagerry[1048974]: Traceback (most recent call last):
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "threading.py", line 980, in _bootstrap_inner
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "cmdb/event_management/event_manager.py", line 321, in run
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "cmdb/event_management/event_manager.py", line 306, in __init_connection
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/blocking_connection.py", line 2468, in queue_declare
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/blocking_connection.py", line 1290, in _flush_output
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/blocking_connection.py", line 458, in _flush_output
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/select_connection.py", line 495, in poll
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/select_connection.py", line 1114, in poll
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/select_connection.py", line 831, in _dispatch_fd_events
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/base_connection.py", line 410, in _handle_events
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/base_connection.py", line 464, in _handle_read
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/connection.py", line 2021, in _on_data_available
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/connection.py", line 2142, in _process_frame
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/connection.py", line 2120, in _process_callbacks
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/callback.py", line 60, in wrapper
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/callback.py", line 92, in wrapper
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/callback.py", line 236, in process
Jun  3 14:57:34 dg1 datagerry[1048974]:  File "pika/adapters/blocking_connection.py", line 1357, in _on_channel_closed
Jun  3 14:57:34 dg1 datagerry[1048974]: pika.exceptions.ChannelClosed: (405, "RESOURCE_LOCKED - cannot obtain exclusive access to locked queue 'datagerry.eventbus.webapp' in vhost '/'. It could be originally declared on another connection or the exclusive property value does not match that of the original declaration.")
Jun  3 14:57:35 dg1 datagerry[1048974]: [2024-06-03 14:57:35][DEBUG   ] --- Gunicorn starting with auto reload option (gunicorn.py)
Jun  3 14:57:35 dg1 datagerry[1048974]: [2024-06-03 14:57:35][INFO    ] --- Interfaces started @ http://0.0.0.0:4000 (gunicorn.py)

I have since stopped and restarted both datagerry and rabbitmq to no avail.

Ideas most welcomed.

Regards,
Kitchen Sink

2

datagerry-postUpgradeRHEL
datagerry-postUpgradeRHEL1114×754 88 KB

Here is the screenshot of the web gui.

3

Hi @sink1 ,
it looks like an issue with rabbitMQ. You should kill the RabbitMQ and DATAGERRY process and then restart them both.

Alternatively check out the RabbitMQ forums regarding the Error: "RESOURCE_LOCKED - cannot obtain exclusive access to locked queue…

What was the API call supposed to do? If it created an object in the database, then delete the created object.

What whas the exact API call which you send, can you post it here ?

In the web GUI press on try to connect and then on the right side Use Connection when the frontend reconnected.

BR Adnan

5

I did press the Try to connect button, but nothing happened.

Regarding the queues:

# for i in $(rabbitmqctl list_vhosts); do echo vhost: $i && rabbitmqctl list_queues -p $i; done
vhost: Listing
Timeout: 60.0 seconds ...
Listing queues for vhost Listing ...
vhost: vhosts
Timeout: 60.0 seconds ...
Listing queues for vhost vhosts ...
vhost: ...
Timeout: 60.0 seconds ...
Listing queues for vhost ... ...
vhost: name
Timeout: 60.0 seconds ...
Listing queues for vhost name ...
vhost: /
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name    messages
datagerry.eventbus.webapp       0
datagerry.eventbus.exportd      0
#

I did restart rabbitmq, but no luck.

However, these URLs do work:

https://dg1.example.com/framework/object/view/24

but not:

https://dg1.example.com/connect

6

Hi,

the API call was:

Invoke-RestMethod -Method Post -Uri ($URL + '/objects/') -Headers $headers -ContentType "application/json" -Body ($vlan | ConvertTo-Json)

… and the variable vlan contained:

$vlan = @{
    "type_id" = 10;
    "version" = "1.0.1";
    "author_id" = 1;
    "active" = $true;
    "fields" = @(
                @{ "name" = "vlan_id"; "value" = 1; };
                @{ "name" = "cidr"; "value" =  "1.1.1.1/24"; }
                )               
    "views" = 0;
}
7

Hi, Problem solved with:


test> use datagerry
switched to db datagerry
datagerry> show collections
datastorage.counter
exportd.jobs
exportd.logs
framework.categories
framework.links
framework.locations
framework.logs
framework.objects
framework.sectionTemplates
framework.types
management.groups
management.users
management.users.settings
settings.conf
datagerry> db.framework.objects.drop()
true
datagerry> show collections
datastorage.counter
exportd.jobs
exportd.logs
framework.categories
framework.links
framework.locations
framework.logs
framework.sectionTemplates
framework.types
management.groups
management.users
management.users.settings
settings.conf
datagerry>

And it all worked again.

8

Nice to hear that :slight_smile:

BR Adnan

9

We think it was this entry that caused the problem in db.framework.objects

{
  _id: ObjectId('665d9811dff7237d6b5f0689'),
  type_id: 10,
  status: null,
  version: '1.0.0',
  creation_time: ISODate('2024-06-03T10:16:49.998Z'),
  author_id: 1,
  last_edit_time: null,
  editor_id: null,
  active: true,
  fields: {
    cidr: '1.1.1.1/24',
    vlan_id: 1
  },

We tried to replicate the condition again, but we have mislaid some of the exact variables used in the api call originally sent :frowning:

10

Hi @sink1,
i think that the fields of the object are wrong. The schema for fields looks like this:
image

So fields is an array of dicts, where the “name”-key contains the identifier and the “value”-key contains the value of the field.

The “_id” property is also not required, MongoDB will create it automatically when the object is inserted.

BR Adnan