An exploratory 3D video game for the Oculus Quest 2, exploring the aesthetics of commodities. With the loss of physical collections, what could be excavated or recreated through machine learning or predictive and visualisation technologies? What would a present-day re-enactment of the journey and re-collection of the tropical imaginary on board the Fame could look like?
“The Collector” is an immersive experience that allows the player to visually explore commodity flows through a modern-day retelling of a shipwreck event. An inspiration for the work is the sinking of the “Fame” in 1824. Hired to return Raffles and his family from Singapore to London, the ship had been a veritable ark of flora and fauna and materials - but the collections were destroyed in a fire, triggering a frenzied attempt to recover the lost tropical imaginary before setting sail again.
Commodities are defined as any bulk good or raw material that has entered into international trade. Commodities are key to the global economy, but their physical production and circulation is often associated with questions of their environmental impact to societies and the world. Today commodities are exchanged without physical goods even exchanging hands – instead futures contracts are bought and sold.
With a hands-on understanding of tropical commodities becoming increasingly out of reach, I wanted to produce an interactive experience where the player could visually explore representations of commodities and the lost tropical imaginary. Using data from “Trading Consequences”, a big data project on international commodities in the British Empire during the 19th century, commodities that flowed in and out of Singapore in the 19th century are re-imagined through google image scrapings and neural style transfer.
“The Collector” is a VR game in two parts. In “The Sorter”, you play as an agent inside an Unforgetting Machine, sorting commodities into nonsensical but serendipitous categories. In “The Collector”, you’re onboard a modern-day Fame, where you can generate an infinite flow of commodities with ease. But beware: if you force the machine to generate more commodities than the boat can hold, the boat will catch fire, and you have no choice but to go under…
A lot of my past research has been into the inventories of objects. I was interested in looking at the manifest of ships which were actively moving cargos (especially that of significant colonial expeditions) in and out of Singapore. One significant expedition that captured my imagination at the time was the disastrous voyage of the Fame in 1824. A company ship had been specially hired to return Sir Thomas Stamford Raffles and his family from Singapore to London, and Raffles had turned the Fame into a veritable ark of flora and fauna, with stacked tiers of boxes of live animals and plants, meticulously collected and documented by Raffles during his long tenure in the East Indies as British salesman, government servant, and keen explorer/naturalist. Tragically, none of his collections were to make it to London. Every single item onboard was destroyed in a fire soon after the boat set sail.
Having lost everything including all of his personal belongings, in what was later to be a failed bid for an insurance claim, Raffles and his wife tried to devise a list of his entire inventory from memory. Raffles wrote a report on the loss of the Fame to the Court of Directors, giving an account on the lost official documentation of the Eastern Isles which would have been of great value to the East India Company, and also added an inventory of his and Sophia’s personal property, including a rather disparate list of items:
Malay manufactures, embroideries and cloths of all kinds and patterns intended as samples for British Industries Drugs and Ivories Lady R’s harp and music Shawls, muslins, curiosities. Family table linen Gold in dust and coinage of the country, of antiquarian significance.Harder to inventorise was the loss of all of his personal papers, correspondences, notes, memos, records of administration, and collection of drawings. He also hired draftsmen and artists to recreate some of the over 2000 maps and natural history illustrations which were lost on the Fame, attempting to recover as much of his collection as possible within 10 weeks before the next ship came to take him home.
Today, the surviving documents which form part of Raffles’ attempts to re-capture the imaginary tropical for a British audience across the seas are now largely stored in the British Library. The challenges that Raffles faced foreground the precarity of colonial expeditions and the risks involved in the endeavour of trying to move a huge quantity of field collections across the ocean.
Current research on the Fame has been largely limited to historical studies (including the historiography of natural history), but I felt there was a need for a more experimental, speculative approach in speaking about lost collections.
To represent the items on The Fame, I extracted a list of 1640 “commodity concepts” traded in Singapore from Trading Consequences data.
Scholars interested in nineteenth-century global economic history face a voluminous historical record. Conventional approaches to primary source research on the economic and environmental implications of globalised commodity flows typically restrict researchers to specific locations or a small handful of commodities. By taking advantage of cutting-edge computational tools, the project was able to address much larger data sets for historical research, and thereby provides historians with the means to develop new data-driven research questions. In particular, this project has demonstrated that text mining techniques applied to tens of thousands of documents about nineteenth-century commodity trading can yield a novel understanding of how economic forces connected distant places all over the globe and how efforts to generate wealth from natural resources impacted on local environments.
The large-scale findings that result from the application of these new methodologies would be barely feasible using conventional research methods. Moreover, the project vividly demonstrates how the digital humanities can benefit from trans-disciplinary collaboration between humanists, computational linguists and information visualisation experts.
Source: From the White Paper on Trading Consequences
From the list of 1640, I manually eliminated the most obvious text errors such as “March April Hay” and “Agar” (which in this case was a repeated misspelling/erroneous keyword identification of Tanjong P-AGAR), resulting in a list of 519 objects.
Many had evocative names like:
Ceylon Madras Gibraltar Malt English Coal Hobart Town Iron Pyrites Jamaica Pepper Johore Sago Macassar Tapioca Manila Hemp New Zealand Timber Penang Pepper Queensland Butter Russian Camel Hair Siam Rice Shanghai Sheep Singapore Flour South Australian Kangaroo Swallow Metal Sydney Guano Tasmania FruitHow would I turn these into icons of themselves, ciphers, without word labels, in this day and age? I downloaded 6 images for each keyword from Google Images. There are ways to automate this process, although some batch renaming and manual checking and cleanup are required.
The Google Images could be used as the base material to create the texture for a sphere shape. I noticed how the images grabbed from Google Images come in different scales: The objects pictured - Some are zoomed in and show the commodity singularly (highlighted in red), Some are zoomed out, and show a group/multiples of the commodity (highlighted in blue). For each keyword that I have downloaded Google Images for, I identified a Content (Singular) and Style (Group/Multiples) and then applied the style transfer on them.
Neural Style Transfer
"Neural style transfer is an optimization technique used to take two images—a content image and a style reference image (such as an artwork by a famous painter)—and blend them together so the output image looks like the content image, but “painted” in the style of the style reference image. This is implemented by optimizing the output image to match the content statistics of the content image and the style statistics of the style reference image. These statistics are extracted from the images using a convolutional network.”
“At a high level, in order for a network to perform image classification (which this network has been trained to do), it must understand the image. This requires taking the raw image as input pixels and building an internal representation that converts the raw image pixels into a complex understanding of the features present within the image."
This is also a reason why convolutional neural networks are able to generalize well: they’re able to capture the invariances and defining features within classes (e.g. cats vs. dogs) that are agnostic to background noise and other nuisances. Thus, somewhere between where the raw image is fed into the model and the output classification label, the model serves as a complex feature extractor. By accessing intermediate layers of the model, you're able to describe the content and style of input images.”
The resulting image is used as the texture for the “currency” token for the material.
The installation features an interactive virtual reality (VR) experience and a simulated gameplay of the journey of discovering the Fame’s collections. Two videos of hands sorting through various artefacts and punch cards build upon the idea of discovering and processing bodies of knowledge and information. Completing the installation is a backdrop featuring fiducials—markers that are a point of reference in the VR world Ding has created. The Collector is an exploration of the interconnection and interdependency of the world through commodities, and reflects on the possibilities machine learning affords in understanding history and constructing bodies of knowledge.