Jacenia LiProjects



Decoding Noises

A machine learning sound project inspired by information and noises using p5.js to send random letter data generated by ultrasonic sensor in Arduino to chatGPT's openAI API.

Concept:

We are cybernetic bodies that carries information, runs around in the world, and interacts with other information bits.

We cannot communicate without interference. In the neverending loops and echoes of information, are messages transmitted and information replicated? Perhaps these unexpected miscommunications are what makes us organic. Data seem certain and permanent, but fleeting noises always exist.

With (de)coding noises, the computer serves as an interpreter of human bodies that transform them into odd phrases and undecipherable noises.

I imagined humans as tiny information bits that runs around and interact with people around the world.

References:
Concept of information and noises from Designs on the Body by Hayles, Network Sovereignty by Marisa Elena Duarte, and the figure about information from A Mathematical Theory of Communication by Shannon
Interactions:

When people approach the installation, arduino detects the distance and sends an api request to chatGPT's openAI api to generate a text response when the person gets really close. And the installation will play sound noises after getting the text response from the p5 sketch.



Machine Learning ----- chatGPT’s openAI API

Live interaction with chatGPT
1. Send random letters to the API
2. Make it interpret them into an odd phrase
3. Output phrase and play sound


Process Documentation


Part 1: Coding - P5.js
  1. create a p5 sketch combining p5 oscillator with ml5 handpose and poseNet machine learning models, as well as text input functions -- https://editor.p5js.org/tl3417/sketches/H5mszX-_Y
  2. deciding on which machine learning model to use
  3. testing out ml5 handpose, poseNet, and openAI api
combine chatGPT’s openAI api into text oscillator
input sound only -> input & output sound

Part 2: Physical Computing - Arduino
  1. test serial communication from arduino to p5js / from p5js to arduino 
  2. tried to work with dc motors for physical output part (decided not to continue this idea)
  3. one button trial
  4. improve interactions to better suit it with my concept — ultrasonic sensors, pulse sensors, and arduino accelerators
  5. use ultrasonic sensors

Part 3: Fabrication - Soldering & Lasercutting