We do not See Until We See

With simple video editing like green screen, or with mirror fragments reflections, or just with a coloured background, Peter Campus took many videos of himself, and he was good at use very simple behaviour (performance/interaction) to make us think what self/ego is, and what the relation between self and other, or self and the surrounding is. Bill Viola’s videos were not himself, but they were also about self reflection. And water, reflection, distortion are some of his favourite materials, or rather symbols in his work. Today there are so many “new media artworks” using camera and self image, but not really having a clear intention (more like a trend or for fun). However, from those early video arts pioneers works, I gained deeper understanding of reflection and self image in an artwork, and I’m inspired to concern: What does it mean to see oneself’s reflection? What’s the different between me and my reflection? Can I really see myself? Do I think about my ego or the other when see my reflection? What do we think while seeing the other’s reflection…

1. 2.
3. 4.
5.

1. Three Transitions (1973), Peter Campus
2. The Reflecting Pool (1979), Bill Viola
3. Interface (1972), Peter Campus
4. Third Tape (1973), Peter Campus
5. The Messenger (1986), Bill Viola

2 - Matter, Phenomenon, Experience

Some people saw the immersiveness (engagement) of the dematerialized experience as an excuse for utopian escapism...But I was, on the contrary, hoping that dematerialization would actually allow you to evaluate a certain set of questions that would otherwise be hard to raise.

...I became very curious about how we might introduce the notion that the person perceiving is not a consumer passively taking in something but, in fact, a producer.

... there was a very strong sense of collective motion a lot of people being physically active in a very non-hierarchical way...I had not foreseen all of them, but the opportunity was still there for me to host the possibility of a recalibration of how body, muscle memory, and cognition actually work together.

Olafur Eliasson (2018)
Excerpt from More than Real, Art in the Digital Age,
chapter ‘But Doesn’t the Body Matter?’

Olafur Eliasson is another significant artist inspiring my project. His works inspired me to use physical material, nature material (like water), and shape it in a simple and nice aesthetic form, however the form itself should be able to tell people to experience it, to perceive it. His installations are like the ineffable phenomenons, the audience don’t need any instruction, any guide, or any tools, they just need to see it, then they would be just attracted by the pieces and step into the pieces, even become parts of the pieces, meanwhile the audience are welcome to just have fun with it, or to reflect on deeper thoughts from it. The material, the form, the design, the audience experience, the deliver meanings, I found it all greatly poetic, and that’s what I would like to create in this (and the further) project(s).

"consciousness occurs as the simultaneity of a conscious act and its object."

# interaction is not about techniques
# no need to explain the audience what to do to interact
# What do you want the audience to feel during the experience?
# What's the difference between to experience an artwork alone VS in company?

# water, dematerialization

Olafur Eliasson
1. The Seeing Space(2015)
2. Big Bang Fountain (2014)

Other related artwork:
Liquid Views - Narcissus' Mirror (1992), Monika Fleischmann

3 - Happening RightNow

The project was also inspired during seeing the drastic protest actions in HongKong against the political conflicts between HongKong and the mainland Chinese government. I was born and grown up in Guangzhou, and my mother’s families are living in HongKong for years. We speak the same language, share the same culture, so could say that there’s a bit more connection between this city and me, comparing with most of the people in mainland China. And I’ve known there’re some political decision conflicts between the two governments and there’re many culture and value conflicts between the people. However, I was more like a spectator, who wasn’t really very concerned about these matters. Until recently people’s anger bringing violence and destroy on the streets, as well as governments’ policy changing, which actually influencing my families and friends’ lives there, then I started to really perceive these issues, look into these incidents. In the end, the more I look into it, the more I see this is not “their matters there”, but “our matters here”.

We might believe the wave was too minor to perceive, until we come forward and look into it, seeing ourselves being inside the water and being affected by the wave, then we actually perceive the wave. - In fact, this behavioural concepts could apply to many objects beyond the HongKong-Mainland matters: politics, social and economic trends, ecosystems, climates, etc. If we don’t step forward and look into it, we would see nothing, and we won’t understand what it is.

Later developed Sketch

The project was built with OpenFrameworks and Arduino. The system includes 4 parts:
(1) ARDUINO controlled PUMPS which generates ripples/waves in the water;
(2) WEBCAM gets motion information of the water waves (using algorithm of background differencing and brightness value calculation), and use the data to generate, or rather to affect the computer graphics on the display screen;
(3) DISPLAY SCREEN shows designed computer graphics combined with the webcam stream of the water and the audience;
(4) AUDIENCE react to the screen and/or the water (mirror), those reactions (e.g. gazing or paying attention to the work) are captured by the webcam, with the algorithm of face tracker, the data then affects the electric current value which goes through the Arduino and pumps to control the water waves generating.

1 - The Mirror and The Camera

“The viewers notice it, look at it, but not be aware of they are parts of the system before they get engaged in the system, or while they are interacting with the system - after the experience, they might (or might not) find out they are parts of the system.” To achieve this very first intention of this project, my first experiment goal was to “hide the camera”. At first, I experimented with the double side mirror, which the dark side of the mirror can see through the bright side of the mirror without being seen. To test it, I tried to place the camera in a black box, then covered the box with the double side mirror. I tested with both the 3mm mirror and the mirror filter sheet. But see-through mirror is not equal to a transparent layer, both of them covered quite a lot light, and the camera image was too dark to get information precisely. Then to solve the lighting problem, I tried to place led light tubes around the mirror surface to light up the captured side, as well as change the webcam from Sony Ps3 Eye to Logitech C920, which the camera settings could be pre-managed by a software called Webcam Settings. However, the result was still not very satisfying.

Then I gave up and decided to just place the webcam above the water tank on the ceiling, or at the top of the screen, and using algorithm like face detection, face tracker, and YOLO object (person) classification to get information like audience amount and audience distance to control the Arduino. Luckily during this experiment process, I found that the webcam could detect human face/person class from the mirror reflection, which could be a good setup to “see the audience without being seen obviously”. So the setup of placing a mirror in the water tank and a webcam capturing image above the tank became the final plan.

2 - How to make sure the audience look into the mirror in a right angle?

My initial plan was to cover a full mirror surface above the pumps (with holes in the mirror for the pumps, and the mirror surface was also underneath the water surface). But from the feedbacks from peers and tutors, I realised that if there’s such a big mirror out there, people were actually not that interested in it, they could just see the mirror in the tank and pass by from a certain distance, without coming forward to look into the tank. And there’s a large possibility that people won’t look into the mirror from the precisely front of the tank, but look into it from the left/right side of it, which would give a wrong angle of face image to the camera, and the face classification algorithm would not work with it. In addition, the face tracker (face classification) algorithm works best with 1 person’s face, but with a large scale mirror, audience with company more tend to look into it together rather than one by one. Therefore after a few tests, I decided to use a small scale mirror in the centre of the tank - which kind of forcing the viewer to step forward and look into it alone. Like Brenda Laurel on Don Norman said in The Design of Everyday Things: "What is represented in the interface is not only the environment and tools but the process of interaction”. As for the shape of mirror, to present the poetic of the installation in terms of aesthetic/form, since there were all straight lines and angles in the plinth, tank, and screen, I decided to use a circle mirror to bring some curves in the setup, which also corresponding to the softness of water.

3 - Arduino and Pumps

To control the pumps with Arduino through data from a webcam in OpenFrameworks, my workflow was : (1) following example from Youtube Easy arduino projects, run pumps with Arduino; (2) following example from keitasumiya, test sending gradient serial value from OpenframeWork to Arduino with simple LED; (3) replace LED lights with pumps in the circuit, and add code of face tracker, using the boolean of if there’s a face as the trigger to increase/decrease the serial value to control the electric current goes through the Arduino.

4 - Shader Computing Graphics

Since from my last Creative Coding workshop project’s experience, I believed that in OpenFrameworks running the visuals by calculating with GPU is necessary, otherwise it would be way too slow to running complicated system and/or sophisticated visuals. So I learned to use shader from Andy Lomas workshop resource and the OpenFrameworks examples. The final visual was created with 2 shaders.

The first one is the background layer, modified from @kyndinfo’s Scaled Noise, I modified some values to make it looks like a streaming waterfall, but in a very computational and glitchy aesthetic. And I combined a texture of live camera image with it, to give the visuals a bit more variation lively. The speed this “waterfall” flows depended on how intense was the water waving in the tank, which was captured by the webcam on top and processed with a background differencing algorithm and calculated from the average brightness value of the differencing images. When the water was calm, the graphic streamed slow, when the water waved intensely, it streamed faster. The second shader is the circle layer. It’s a webcam live video capturing from the top of the tank, cropped with the WebcamSetting software, then used a circle-in-rectangle image mask to cropped again in OpenFramework to only draw the circle mirror area. After that, I put this camera image as a texture of a noise distortion shader modified from @patriciogv_Fractal's Brwonian Motion. And designed it to look more or less dissolved in the background, then only became clearer when there’s a viewer stepped forward, looking into the mirror, and turned up the water waves - which was captured by the webcam background differencing.

Ólafur Elíasson, Godfrey, M., Tate Modern, Solomon R. Guggenheim Museum, 2019. Olafur Eliasson: in real life.

Barlow, M., 2004. Touch: Sensuous Theory and Muitisensory Media Laura U. Marks: Touch: Sensuous Theory and Muitisensory Media Laura U. Marks Minneapolis: University of Minnesota Press, 2002, 259 pp. Canadian Journal of Film Studies 12, 126–129. https://doi.org/10.3138/cjfs.12.2.126

Daniel Birnbaum editor, Michelle Y. Kuo editor, 2018. More than real: art in the digital age : 2018 Verbier Art Summit. Koenig Books, London.

John. McCarthy, 2004. Technology as experience. MIT Press, Cambridge, Mass.

DIY MINI FOUNTAIN SHOW LIKE DUBAI - ARDUINO, n.d.

Sumiya, K., 2017. keitasumiya/oF_arduino.

Shaders: Common ingredients for Fragment Shaders [WWW Document], n.d. URL https://learn.gold.ac.uk/mod/book/view.php?id=652211&chapterid=18156.

GLSL Shader Editor [WWW Document], n.d. . The Book of Shaders. URL https://thebookofshaders.com/edit.php?log=161127200614.

GLSL Shader Editor [WWW Document], n.d. . The Book of Shaders. URL https://thebookofshaders.com/edit.php#11/lava-lamp.frag.

ofBook - Introducing Shaders [WWW Document], n.d. URL https://openframeworks.cc/ofBook/chapters/shaders.html.