dernière modification : 2023

JSON et Unix

Certaines commandes Unix (de plus en plus) peuvent sortir leurs résultats au format JSON (par exemple tree, journalctl, lsblk, ip). D’autres requierent une configuration en JSON (nmcli, par exemple).

Il est donc utile de savoir traiter rapidement des données JSON sans passer forcément par de la programmation.

1. L’exemple de lsblk

La commande unix lsblk liste les périphériques de la machine :

$ lsblk
NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
nvme0n1                                       259:0    0  477G  0 disk
├─nvme0n1p1                                   259:1    0    2G  0 part  /boot
└─nvme0n1p2                                   259:2    0  475G  0 part
  └─luks-7bf4f0c8-ca29-4f68-9b6e-069db5a6b326 253:0    0  475G  0 crypt
    ├─vg-ro2                                  253:1    0   50G  0 lvm   /
    ├─vg-swp                                  253:2    0  7,6G  0 lvm   [SWAP]
    ├─vg-tmp                                  253:3    0    2G  0 lvm   /tmp
    ├─vg-ro1                                  253:4    0 48,8G  0 lvm
    └─vg-dat                                  253:5    0   50G  0 lvm   /data

Pour récupérer une information précise dans cette sortie texte, il faut parser la sortie. Unix dispose de toutes les commandes de traitement de texte pour le faire (sed, awk, etc.), mais cela reste dépendant du formatage de la sortie, en espérant que celui-ci ne varie pas avec la langue ou les versions successives de la commande .

Une sortie au format JSON permet de rigidifier la sortie :

$ lsblk -J
{
   "blockdevices": [
      {"name":"nvme0n1", "maj:min":"259:0", "rm":false, "size":"477G", "ro":false, "type":"disk", "mountpoint":null,
         "children": [
            {"name":"nvme0n1p1", "maj:min":"259:1", "rm":false, "size":"2G", "ro":false, "type":"part", "mountpoint":"/boot"},
            {"name":"nvme0n1p2", "maj:min":"259:2", "rm":false, "size":"475G", "ro":false, "type":"part", "mountpoint":null,
               "children": [
                  {"name":"luks-7bf4f0c8-ca29-4f68-9b6e-069db5a6b326", "maj:min":"253:0", "rm":false, "size":"475G", "ro":false, "type":"crypt", "mountpoint":null,
                     "children": [
                        {"name":"vg-ro2", "maj:min":"253:1", "rm":false, "size":"50G", "ro":false, "type":"lvm", "mountpoint":"/"},
                        {"name":"vg-swp", "maj:min":"253:2", "rm":false, "size":"7,6G", "ro":false, "type":"lvm", "mountpoint":"[SWAP]"},
                        {"name":"vg-tmp", "maj:min":"253:3", "rm":false, "size":"2G", "ro":false, "type":"lvm", "mountpoint":"/tmp"},
                        {"name":"vg-ro1", "maj:min":"253:4", "rm":false, "size":"48,8G", "ro":false, "type":"lvm", "mountpoint":null},
                        {"name":"vg-dat", "maj:min":"253:5", "rm":false, "size":"50G", "ro":false, "type":"lvm", "mountpoint":"/data"}
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

La commande jq reformate plus lisiblement un JSON. Exemple avec la commande journalctl :

sortie plain text
$ journalctl
mars 03 20:47:38 pc-jaclin kernel: nvme 0000:02:00.0:    [ 0] RxErr                  (First)
mars 03 20:47:38 pc-jaclin kernel: nvme 0000:02:00.0:   device [1c5c:174a] error status/mask=00000001/00000000
mars 03 20:47:38 pc-jaclin kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
mars 03 20:47:38 pc-jaclin kernel: pcieport 0000:00:1b.0: AER: Corrected error received: 0000:02:00.0
sortie JSON
$ journalctl -o json
{"_MACHINE_ID":"2d7c653c4ac94e65921bd7ccc1440bc9","SYSLOG_FACILITY":"0","_KERNEL_SUBSYSTEM":"pci","__MONOTONIC_TIMESTAMP":"259142405735","_SOURCE_MONOTONIC_TIMESTAMP":"259139129475","_KERNEL_DEVICE":"+pci:0000:02:00.0","SYSLOG_IDENTIFIER":"kernel","_HOSTNAME":"pc-jaclin","_TRANSPORT":"kernel","MESSAGE":"nvme 0000:02:00.0:    [ 0] RxErr                  (First)","__REALTIME_TIMESTAMP":"1677872858772770","__CURSOR":"s=2ff966ef3bfa4e619e4fd79a86ab8f5b;i=3a7f;b=75ddd32912244679a623baf6ff9a7249;m=3c56174e67;t=5f60437b9b522;x=a80de7dda6996a45","_BOOT_ID":"75ddd32912244679a623baf6ff9a7249","_UDEV_SYSNAME":"0000:02:00.0","PRIORITY":"4"}
{"SYSLOG_IDENTIFIER":"kernel","_TRANSPORT":"kernel","_UDEV_SYSNAME":"0000:02:00.0","__CURSOR":"s=2ff966ef3bfa4e619e4fd79a86ab8f5b;i=3a7e;b=75ddd32912244679a623baf6ff9a7249;m=3c56174c6b;t=5f60437b9b326;x=1cfe88cc53854e9","_KERNEL_DEVICE":"+pci:0000:02:00.0","MESSAGE":"nvme 0000:02:00.0:   device [1c5c:174a] error status/mask=00000001/00000000","_MACHINE_ID":"2d7c653c4ac94e65921bd7ccc1440bc9","SYSLOG_FACILITY":"0","__MONOTONIC_TIMESTAMP":"259142405227","_KERNEL_SUBSYSTEM":"pci","_BOOT_ID":"75ddd32912244679a623baf6ff9a7249","__REALTIME_TIMESTAMP":"1677872858772262","PRIORITY":"4","_SOURCE_MONOTONIC_TIMESTAMP":"259139129467","_HOSTNAME":"pc-jaclin"}
{"__REALTIME_TIMESTAMP":"1677872858771690","_BOOT_ID":"75ddd32912244679a623baf6ff9a7249","_SOURCE_MONOTONIC_TIMESTAMP":"259139129460","MESSAGE":"nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)","_UDEV_SYSNAME":"0000:02:00.0","_KERNEL_SUBSYSTEM":"pci","SYSLOG_FACILITY":"0","__CURSOR":"s=2ff966ef3bfa4e619e4fd79a86ab8f5b;i=3a7d;b=75ddd32912244679a623baf6ff9a7249;m=3c56174a2f;t=5f60437b9b0ea;x=e6f263226938fe13","_MACHINE_ID":"2d7c653c4ac94e65921bd7ccc1440bc9","_HOSTNAME":"pc-jaclin","_KERNEL_DEVICE":"+pci:0000:02:00.0","SYSLOG_IDENTIFIER":"kernel","PRIORITY":"4","__MONOTONIC_TIMESTAMP":"259142404655","_TRANSPORT":"kernel"}
{"SYSLOG_IDENTIFIER":"kernel","_UDEV_SYSNAME":"0000:00:1b.0","_SOURCE_MONOTONIC_TIMESTAMP":"259139129432","SYSLOG_FACILITY":"0","_MACHINE_ID":"2d7c653c4ac94e65921bd7ccc1440bc9","PRIORITY":"6","_HOSTNAME":"pc-jaclin","__CURSOR":"s=2ff966ef3bfa4e619e4fd79a86ab8f5b;i=3a7c;b=75ddd32912244679a623baf6ff9a7249;m=3c561746fa;t=5f60437b9adb3;x=8530ef8fb5a9309a","__MONOTONIC_TIMESTAMP":"259142403834","_KERNEL_DEVICE":"+pci:0000:00:1b.0","_TRANSPORT":"kernel","MESSAGE":"pcieport 0000:00:1b.0: AER: Corrected error received: 0000:02:00.0","_BOOT_ID":"75ddd32912244679a623baf6ff9a7249","_KERNEL_SUBSYSTEM":"pci","__REALTIME_TIMESTAMP":"1677872858770867"}
formatage avec jq
$ journalctl -o json | jq
{
  "MESSAGE": "nvme 0000:02:00.0:    [ 0] RxErr                  (First)",
  "PRIORITY": "4",
  "_UDEV_SYSNAME": "0000:02:00.0",
  "_SOURCE_MONOTONIC_TIMESTAMP": "259139129475",
  "_HOSTNAME": "pc-jaclin",
  "SYSLOG_FACILITY": "0",
  "_BOOT_ID": "75ddd32912244679a623baf6ff9a7249",
  "__REALTIME_TIMESTAMP": "1677872858772770",
  "__MONOTONIC_TIMESTAMP": "259142405735",
  "__CURSOR": "s=2ff966ef3bfa4e619e4fd79a86ab8f5b;i=3a7f;b=75ddd32912244679a623baf6ff9a7249;m=3c56174e67;t=5f60437b9b522;x=a80de7dda6996a45",
  "_KERNEL_DEVICE": "+pci:0000:02:00.0",
  "SYSLOG_IDENTIFIER": "kernel",
  "_KERNEL_SUBSYSTEM": "pci",
  "_MACHINE_ID": "2d7c653c4ac94e65921bd7ccc1440bc9",
  "_TRANSPORT": "kernel"
}
{
  "__MONOTONIC_TIMESTAMP": "259142405227",
  "_HOSTNAME": "pc-jaclin",
  "_UDEV_SYSNAME": "0000:02:00.0",
  "__CURSOR": "s=2ff966ef3bfa4e619e4fd79a86ab8f5b;i=3a7e;b=75ddd32912244679a623baf6ff9a7249;m=3c56174c6b;t=5f60437b9b326;x=1cfe88cc53854e9",
  "_KERNEL_SUBSYSTEM": "pci",
  "_MACHINE_ID": "2d7c653c4ac94e65921bd7ccc1440bc9",
  "_TRANSPORT": "kernel",
  "SYSLOG_FACILITY": "0",
  "MESSAGE": "nvme 0000:02:00.0:   device [1c5c:174a] error status/mask=00000001/00000000",
  "PRIORITY": "4",
  "SYSLOG_IDENTIFIER": "kernel",
  "__REALTIME_TIMESTAMP": "1677872858772262",
  "_SOURCE_MONOTONIC_TIMESTAMP": "259139129467",
  "_KERNEL_DEVICE": "+pci:0000:02:00.0",
  "_BOOT_ID": "75ddd32912244679a623baf6ff9a7249"
}
...

2. jo

Générer du JSON en shell est laborieux :

$ echo '{"name":"Jane"}'
$ echo '{ "Maná": { "Fernando Olvera": { "role": [ "vocals", "guitar", "harmonica" ], "entrance": 1981, "exit": null }, "Alejandro González": { "role": [ "drums", "backup vocals", "percussions" ], "entrance": 1984, "exit": null }, "Juan Calleros": { "role": [ "bass guitar" ], "entrance": 1981, "exit": null }, "Sergio Vallin": { "role": [ "guitar", "backup vocals" ], "entrance": 1995, "exit": null } }, "Santana": { "Carlos Santana": { "role": [ "lead guitar" ], "entrance": 1966, "exit": null }, "Tom Fraser": { "role": [ "rythmic guitar" ], "entrance": 1966, "exit": 1970 }, "Mike Carabello": { "role": [ "percussion" ], "entrance": 1966, "exit": 1971 }, "Keith Jones": { "role": [ "bass guitar" ], "entrance": 1983, "exit": 1989 }, "Michael Shrieve": { "role": [ "drums" ], "entrance": 1969, "exit": 1988 } } }'

jo simplifie l’opératon :

$ jo name=Jane                         # {"name":"Jane"}
$ jo time=$(date +%s) dir=$HOME        # {"time":1457195712,"dir":"/users/jaclin"}
$ jo -p -a spring summer winter
[
   "spring",
   "summer",
   "winter"
]
$ jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17) sunday=false
{
   "name": "JP",
   "object": {
      "fruit": "Orange",
      "point": {
         "x": 10,
         "y": 20
      },
      "number": 17
   },
   "sunday": false
}
$ jo -p number=17 pass=true geo[lon]=88 geo[cc]=ES point[]=1 point[]=2 geo[lat]=123.45
{
   "number": 17,
   "pass": true,
   "geo": {
      "lon": 88,
      "cc": "ES",
      "lat": 123.45
   },
   "point": [
      1,
      2
   ]
}

3. jq

jq est un processeur JSON. Il est à JSON ce qu’est xpath à XML.

Soit le JSON :

jdata='{ "Maná": { "Fernando Olvera": { "role": [ "vocals", "guitar", "harmonica" ], "entrance": 1981, "exit": null }, "Alejandro González": { "role": [ "drums", "backup vocals", "percussions" ], "entrance": 1984, "exit": null }, "Juan Calleros": { "role": [ "bass guitar" ], "entrance": 1981, "exit": null }, "Sergio Vallin": { "role": [ "guitar", "backup vocals" ], "entrance": 1995, "exit": null } }, "Santana": { "Carlos Santana": { "role": [ "lead guitar" ], "entrance": 1966, "exit": null }, "Tom Fraser": { "role": [ "rythmic guitar" ], "entrance": 1966, "exit": 1970 }, "Mike Carabello": { "role": [ "percussion" ], "entrance": 1966, "exit": 1971 }, "Keith Jones": { "role": [ "bass guitar" ], "entrance": 1983, "exit": 1989 }, "Michael Shrieve": { "role": [ "drums" ], "entrance": 1969, "exit": 1988 } } }'

3.1. Formatage

echo $jdata | jq "."
curl http://api.open-notify.org/iss-now.json | jq .

3.2. Chemin

En donnant un chemin à jq, on obtient la valeur (joliment formatée) de l’entité correspondant au chemin.

'.' est la racine de la structure de données, ce qui explique le résultat précédent.

On progresse dans la structure avec le '.' :

echo $jdata | jq '.Santana'
echo $jdata | jq '."Maná"."Juan Calleros"'     (1)
echo $jdata | jq '.Santana."Keith Jones".role' (1)
1 guillemets/quotes nécessaires en présence d’espaces ou de caractères non ASCII

'.Santana' peut s’écrire aussi '.["Santana"]'. De même, '."Maná"."Juan Calleros"' peut s’écrire aussi :

'.["Maná"]."Juan Calleros"'
'."Maná"["Juan Calleros"]'
'.["Maná"]["Juan Calleros"]'
sélection d’un élément de liste
jq '."Maná"."Fernando Olvera".role'        # une liste
jq '."Maná"."Fernando Olvera".role[1]'     # un élement

3.3. Fonctions

echo $jdata | jq 'keys '
echo $jdata | jq '.Santana | keys'
echo $jdata | jq '.Santana | keys | length'

3.4. Itérations

boucle sur une liste
echo $jdata | jq '."Maná"[]'
echo $jdata | jq '. | keys[] '
echo $jdata | jq '."Maná"[].entrance'
echo $jdata | jq '.[][].entrance'
boucle sur un objet
echo $jdata | jq 'keys[] as $k | { ($k) : 1} '
echo $jdata | jq 'keys[] as $k | { ($k) : (.[$k]) | keys} '

3.5. Créer des entités

générer des listes et dictionnaires
echo $jdata | jq '[.[][] | { entree: .entrance }]'