Category: bash

JSON text filtering with jq

14 March 202314 March 2023 Fabien bash, Linux Leave a comment

I frequently have to intereact with some json files , since bash is the main tool i use day to day i must master the usage of jq, but i noticed that the vast majority of totorials on jq usage just sucks.
I’m going to write some kind of memo to remember the main command that i use most of the time
lets take this dummy contact.json file

{
  "contacts": [
    {
      "name": "John Smith",
      "phone": "(123) 456-7890",
      "email": "john.smith@example.com",
      "sex": "male",
      "position": "Manager",
      "birthday": "1985-06-23",
      "devices": [
        {
          "type": "laptop",
          "model": "MacBook Pro",
          "serial_number": "ABCD1234",
          "issued_date": "2022-01-15"
        },
        {
          "type": "phone",
          "model": "iPhone 12",
          "serial_number": "EFGH5678",
          "issued_date": "2022-01-15"
        }
      ]
    },
    {
      "name": "Jane Doe",
      "phone": "(234) 567-8901",
      "email": "jane.doe@example.com",
      "sex": "female",
      "position": "Sales Representative",
      "birthday": "1990-02-14",
      "devices": [
        {
          "type": "laptop",
          "model": "Dell XPS 13",
          "serial_number": "IJKL9012",
          "issued_date": "2022-03-01"
        }
      ]
    },
    {
      "name": "Bob Johnson",
      "phone": "(345) 678-9012",
      "email": "bob.johnson@example.com",
      "sex": "male",
      "position": "IT Specialist",
      "birthday": "1982-11-30",
      "devices": [
        {
          "type": "laptop",
          "model": "ThinkPad X1 Carbon",
          "serial_number": "MNOP3456",
          "issued_date": "2022-02-10"
        },
        {
          "type": "phone",
          "model": "Samsung Galaxy S21",
          "serial_number": "QRST7890",
          "issued_date": "2022-02-10"
        }
      ]
    }
  ]
}

to list the name of all the male employes in the json file
cat contact.json | jq '.contacts[] | select(.sex=="male") | .name'

this one is going to list all the phone and their issued date
cat contact.json | jq '.contacts[].devices[] | select(.type=="phone") | {model: .model, issued_date: .issued_date}'

To display the devices issued after 2022-02-12 and their owner’s name, you can use the following jq command:
cat contact.json | jq '.contacts[] | {owner: .name, devices: [.devices[] | select(.issued_date > "2022-02-12") ] } | select(.devices != [])'

Using bash and gatttool to get readings from Xiaomi Mijia LYWSD03MMC Temperature Humidity sensor

23 January 20205 November 2020 Fabien bash, Linux 21 Comments

There is a new inexpensive Temperature and humidity sensor by xiaomi.
This time is no longer round ,

Xiaomi Mijia LYWSD03MMC Bluetooth 4.2 Temperature Humidity sensor

if you like me would like to get the temperature and humidity data from time to time to import in a graphing tool like grafana, there is a simple solution using classic bash tools and gatttool.
First you have to indentify the mac-address of your little sensor, for this , just make sure the sensor is in range of your linux device ans launch a
hcitool lescann

this command will spit out all the bluetooth devices in range
just find the line with the name of the device and copy the mac address A4:C1:38:8C:77:CA LYWSD03MMC

then you must start a little bash script like this :

#!/bin/bash bt=$(timeout 15 gatttool -b A4:C1:38:8C:77:CA --char-write-req --handle='0x0038' --value="0100" --listen) if [ -z "$bt" ] then echo "The reading failed" else echo "Got data" echo $bt temphexa=$(echo $bt | awk -F ' ' '{print $12$11}'| tr [:lower:] [:upper:] )
humhexa=$(echo $bt | awk -F ' ' '{print $13}'| tr [:lower:] [:upper:])
temperature100=$(echo "ibase=16; $temphexa" | bc)
humidity=$(echo "ibase=16; $humhexa" | bc)
echo "scale=2;$temperature100/100"|bc
echo $humidity fi

this is a skeleton that you can improve , but for now it pretty much work like that ,

first it use gatttol to connect to the sensor and listen for 15 sec
During these fifteen seconds , you can be pretty much sure to receive at least some data like this :

Characteristic value was written successfully Notification handle = 0x0036 value: 58 03 47 a0 0b Notification handle = 0x0036 value: 55 03 47 a0 0b

this tell me that during the 15 sec of connection i received the information that i need two times.
what i’m after are the value 58 03 for the temperature , and 47 for the humidity.

the temperature is little endian format its mean that the two group must be inverted before decoding the data. i invert the values with awk and decode them using bc.
bc doesn’t like when the hex values are not capitalised so tr is used to do that.

bc then give you the temperature multiplied by 100. relaunch bc to divice per 100,

For the humidity its simpler , you get the value in one step without inverting anyting

Then you can do what you need with theses two vars , insert then in some database etc ..

there is room from improvement: for example this script is not capable of decoding negative temperature.
i will post an improved version when i figure out how to do it

Test the existence of a file in bash

11 July 201923 January 2020 Fabien bash Leave a comment

If you need to test in a bash script if a file exist you can launch the following command.

if [ -f /tmp/dnsmasq_leases.log ];then
         echo "The file exist"
 else
        echo "The file doesn't exist"
 fi

Note if the file exist , but , is empty this also test true and tell you that the file exist

space char directly after var in bash

11 July 201923 January 2020 Fabien bash, Linux Leave a comment

In bash space is used to determine the end of a variable ,

for example when you launch

var = abcd
echo $var ef

this will display abcd ef ,if you try to to display abcdef you can‘t do echo $varef

the solution to this little problem is : is to use curly brackets.

var = abcd
echo ${var}ef

will display abcdef

bash : Random number in OpenWRT

27 May 201920 July 2019 Fabien bash, Linux Leave a comment

On some light linux environement running busybox or openWRT you will notice that you don’t have access to the $RANDOM variable ,
the date command is also incomplete so you can’t use the tail of the nanosecond date to have a random number.

grep -m1 -ao '[0-6][0-9]' /dev/urandom | head -n1

Using this command you will get a random number beetween 0 and 69.
if you wish to change the range of the random numbers , just edit the regex in the grep part

Using AWK to display the min the max and the average of a list

28 November 201828 November 2018 Fabien bash, Linux 1 Comment

awk is a extremely powerful tool !
i’m always surprised how much you can do with it.

My latest finding is pretty amazing.

Lets say you have a list of numbers , and you want to figure out the bigger number , the lower number , and the average of all the numbers in the list.

cat tmp.txt
40
10
25
35
11
2.2
99.5
100.1
1.5
5

you can use the following awk code to print minimum, maximum and average.

cat tmp.txt | awk '{if(min==""){min=max=$1}; if($1>max) {max=$1}; if($1<min) {min=$1}; total+=$1; count+=1} END {print "avg " total/count," | max "max," | min " min}'

The output is going to be :
avg 32.93 | max 100.1 | min 1.5

you can also use this code with decimal numbers

Bash – Loop over the content of a file line by line

20 June 201810 October 2018 Fabien bash, Linux Leave a comment

If you need to iterate over the content of a file line by line
You can use this little bash trick.

while read -r line; do
printf '%s\n' "$line"
done < "file.txt"

In this case , it will simply print the line, but if you need to do an operation on the text , just insert your code into the while loop.

Use the keyboard on a remote machine over SSH?

22 April 201822 April 2018 Fabien bash, Linux Leave a comment

If you use a Linux box to display some infos in a public space , you might not have the possibility to connect a mouse and keyboard every time you need to click on the page.

For example, at work we use a TV to display some metrics with grafana,
Sometime, when we reboot the machine , the identification cookie will be outdated and the browser will land on the authentication page instead.
Since the machine has no keyboard and mousse attached this cause some issues , .
So I’ve used the tool xdotool that allow a privileged ssh user or a script to interact with the mouse and keyboard

An important step is to select the screen on witch you want to send mouses clicks and keystrokes.

export DISPLAY=:0

After that lets say your screen is displaying a full-screen browser with a login page ,
By default the cursor is already in the login form.

for the example , the login/password are going to be admin/admin

#for typing the login
xdotool key a && xdotool key d && xdotool key m && xdotool key i && xdotool key n

#to advance to the password form
xdotool key Tab

#for typing admin again in the password prompt.
xdotool key a && xdotool key d && xdotool key m && xdotool key i && xdotool key n

#then tab gain to go the login button , and push Enter to send the form

xdotool key Tab && xdotool key KP_Enter

you are now logged and you dashboard will display

Bash keyboard shortcuts

15 February 201815 February 2018 Fabien bash, Linux Leave a comment

When you are spending your day in the terminal , you will find that navigating using only the arrows keys although fine at the the start will frustrate you because it’s quite slow.
Bash come with a lot of keyboard shortcuts design to gain precious time.
CTRL+A # will move the cursour to the beginning of the line CTRL+E # moves to end of line CTRL+C # halts the current command and back to prompt CTRL+D # deletes one character backward or logs out of current session, similar to exit CTRL+K # deletes all the text forward to the end of the line CTRL+L # clears screen and redisplay the line CTRL+R # searches history entering keyword , CTRL+T # transposes two characters CTRL+U # kills backward from point to the beginning of line CTRL+W # kills the word behind the cursor

i’ve put in bold the shortcuts that i use every day , i don’t use CTRL+A and CTRL+E because the keyboard has keys that are foing the same thing.

Bash brace expansion. (to delete files)

3 February 2018 Fabien bash, Linux Leave a comment

In bash , it is common to have to do some action on a numéric serie of files , like deleting or renaming ,

Using brace expansion you will be able to générate a single line that will act on multiple targets ,

For our example , let’s think about a list of ten files

log1.log log2.log log3.log log4.log log5.log
log6.log log7.log log8.log log9.log log10.log

in the simple case where i want to delete the complete list of files, i just have to run de command,

rm log*.log

The 10 files will be deleted, but , this is not practical if you want to keep the last file , and delete the other nine,

In that case we can use a usefull tool called brace expansion,

To delete the files 1 to 9 , i just have to run in my bash terminal!

rm log{1..9}.log

This last one is extremely practical , I use it very often in a lot of different uses cases.

But in other cases , you may want to delete even on odd files , or one every three files,

This is also doable with braces expansions

rm log{1..10..2} .log
#to delete  1 3 5 7 9
rm log{1..10..3}.log
#to delete 1 4 7 10