Getting Started with TUIO, Reactivision, AS3, and PD

IMG_5332

Recently, a bunch of us were at Universidad Autónoma del Estado de México (Facultad del Artes) in Toluca, teaching a workshop and having an expocision there! Luis gave a class on Blender, Anne-marie taught about Gamemaker, Andreas taught about Pure Data, and I attempted to teach about TUIO/Kinect/Reactivision.

I say attempted, because I and Andreas did not speak much spanish (or rather we did not actually speak ANY spanish on arrival), and as I am a particularly longwinded and wordy person (with some incredibly longwinded and complicated ideas to communicate about the purpose and meaning behind interaction or programming), there were many things lost in translation due to the language barrier, despite the very kind attempts by some of the MA students such as Julio and Blanca (who tried to bridge the language and learning gaps for us by teaching the younger students a little bit more about flash and programming before we got there). But I hope everyone there still learnt something from my murmurings about tangible touch (and thanks to Janitzio for having us all there to teach as well!).

P5254497

IMG_5440

P5254505

P5244488

Some pictures from the part of the workshop that Andreas and I were teaching

The class was actually very sweet and it was lovely to go there and teach them all for a week, although there were truly some moments of utter exasperation of not being able to communicate an idea better as I simply did not have any common words to do so. In the end I also had to improvise and turn the class into a reactivation focused lesson because there were too few kinect sets in the classroom (I had only brought two from Singapore). I suppose I too have learnt a lot of things from this experience:

  1. it is truly very difficult to really teach programming to people who speak a different language to one’s own (especially when simple pseudocode examples do not make any sense to them)
  2. its easier to teach a short workshop with opensource tools like pure data, rather than a program like flash that requires a licence, or with external hardware like kinect, because one really needs to have a day in advance to do a pre-setup session or installfest. i had not catered for this earlier unfortunately. also, that part about Flash files not being readable by more than the version before it (ie: Flash CS5 files are only editable in CS4 and CS5, but not CS3) really bothers me. (caveat: despite this drawback, i feel Flash is a pretty practical tool with its own commercial uses so i would still recommend learning Flash although its a little more bothersome…)
  3. it is very difficult to learn how to make a tangible touch setup AND to write the program if one does not have any basic understanding or grounding in programming. IN FACT LET’S JUST SAY ITS IMPOSSIBLE. this is not going to be the easiest “MY FIRST FLASH APP” to start with. perhaps i could eventually find a way to write a simpler tutorial, but at this point i suppose i don’t have enough teaching experience to know how to make it understandable to someone with TOTALLY ZERO programming experience. it would take much longer. but i suppose maybe as i slowly accrue more teaching experience, i might also learn the mystical way of magically teaching anything to people who have zero knowledge of the thing that you are teaching about……

P5254502

Julio and Blanca in a funny moment. The MA students really helped translate and communicate some of the things I could not say in Spanish to the rest of the class. Thank you so much guys!


P5244490

at the workshop i tried to show examples in both as3 and pure data. having played a little bit with max and pd before, it was straightforward for me to figure out how i could start using pd and tuio from the examples provided although this was also my first time really using PD.

I am by no means some sort of ultimate expert on learning about TUIO (as these were things I had slowly figured out by myself on my own from reading on the internet and posting on forums) but I suppose one thing I’ve appreciated about this in the few years that I have been trying to figure it out is how TUIO is easy and consistent so that when you code up something, say for Flash AS3, you would then have a consistent way to refer to things and a flash app could be built for a touch table OR also used for kinect and it would still work immediately both ways for both setups (assuming you used the term TUIOCursor and TUIOObject consistently)

The difference between TUIOCursor is that a cursor is like a dot on the screen that is tracked and does not have rotational angle. There can be many many Cursors as well, whereas an TUIOObject is something that is already defined an object number and it is an object which has an “Up” or “Down” while on screen and can be rotated around and around. (There is also a TUIOBlob, but suffice to say since its been all pretty homebrew I rarely get around to calibrating my blobs)


Click here to view the full
TUIO + Kinect/Reactivision + AS3 Workshop Guide
http://dreamsyntax.org/tuioworkshop

– This guide is for Mac Users who use 10.5 and above.
– Windows users can also use the guide but some instructions may differ (such as the way one uses the Terminal/Console)
– Example start file available at http://dreamsyntax.org/tuioworkshop/tuioclientexample.zip
– Solution file available at http://dreamsyntax.org/tuioworkshop/tuioclientexample_solution.zip
– Presentation built with slidedown


BONUS! – Getting started with TUIO and PD

To get started with TUIO and PD, download PD Extended from the puredata websitew.

Download the TUIO Client Reference Implementations for PD

Download reactivision and print out some fiducial markers to wave at your camera to test it

Open the programs in this order: reactivision > pure data > the tuio pd patch of your choice, and voila it should work out of the box when you wave a symbol at your webcam!

Screen shot 2012-05-25 at PM 01

The Near Impossibility of Compatibility between a Mac and the Samsung Galaxy Tab 10.1 and Samsung Pico Projector

I’ve been playing around with two devices recently, for work reasons. Both of them are incidentally Samsung devices – namely the Samsung Pico Projector SP-H03 (SGD399 Promo at EpiLife) and Samsung Galaxy Tab 10.1 (Retail Price SGD848). One thing I noticed is that although they are kinda decent products on their own, unfortunately they won’t work right out of the box with a Mac. In fact they pretty much don’t respond at all when connected to a Mac! It’s times like these that I think that Apple does it the right way when I don’t have to install a driver to print on most common printers. In this day and age, we expect the devices to be smart enough to connect up. At the very least I would expect some distant flicker of recognition or reaction from it!

These are my workarounds for connecting them to a Mac, I hope it helps someone else connect their devices to their Mac since it took me some time to figure these out. For the record, I’m using a Macbook Pro 15″ running on Mac OSX 10.6.8.


My issues with both Samsung products:

1. When you plug a mobile device into a computer, you would expect it to mount and charge, like an iPad would on any device whether Mac or Windows. Anything that requires me to install a driver/application and go through a very complicated method to access files shouldn’t have been shipped out to unwitting consumers.

2. When you plug a device such as a projector into a computer, you would expect it to be detectable without having to completely stop what you’re doing and reboot your computer.


 

How to Connect Samsung Galaxy Tab 10.1 to a Mac

I tried using Kies, Android File Transfer, and via DDMS in Android SDK. DDMS stands for Dalvik Debug Monitor Server and its Android’s debugging tool. Kies and Android File Transfer did not work AT ALL with Mac. Android SDK was a bit more hassle to install but essentially straightforward to do and most importantly, at least it works!

Screen shot 2012-05-05 at AM 12
Hopeless Kies: Infinitely spinning “Connecting…” Dialog

Method 1: Kies – FAILED

Will not connect the Galaxy Tab to my Macbook Pro via USB or Wifi. In any case, it could only transfer photos, videos, and movies, and would not have allowed any access to the file structure. It perpetually says “Connecting…” but never does.

Method 2: Android File Transfer – FAILED

Very disappointing as this was the “official” method listed in the Samsung website. For explanation, read this: “Google provide Android File Transfer and there is also a third party product called XNJB. Neither currently work and the reason for this is that they both use libmtp which is an open source implementation of an MTP Initiator. To understand why this is we need to go back to MTP. In MTP each device has a manufacturer code and device code, a list of which is maintained in the libmtp source code (take a look at music-players.h). Building the library hard-wires the device list inside it, which, at the last build of both Android File Transfer and XNJB did not include the requisite codes for the Samsung Galaxy Tab.” – Who made the decision to direct consumers to download AFT if in the first place it is certainly not going to work with the Galaxy Tab? Also, why did they not ship Kies together with the product if its their software?

Screen shot 2012-05-05 at AM 02
Also hopeless: Android File Transfer does not work in this case

Method 3: Via Android SDK – WORKS!

This was the last ditch attempt, but this method was the only thing that worked for me. You’ll want to make sure that you have Eclipse Classic (Mac OS X for 64 bit) + ADT Plugin for Eclipse + Android SDK. I followed this guide on how to install Android SDK in Eclipse and it was a breeze.

Screen shot 2012-05-05 at AM 02
Eclipse

Screen shot 2012-05-05 at AM 02

Installing Android SDK

Screen shot 2012-05-05 at AM 02

Installing ADT Plugin

While the Galaxy Tab is not connected to the computer, go to Applications > USB Debugging and make sure it is ticked so that Debugging mode will launch when the USB is next connected.

Go to the Android SDK toolkit window in Eclipse. Go to Windows > Open Perspective > Others… > DDMS.

Screen shot 2012-05-05 at AM 02
SUCCESS!!! Here’s looking at the files inside the Galaxy Tab 10.1
Now you’re in. But should we have to go to so much trouble to access the device??? I am doubtful that most will take the trouble to do this. Plus its not very pretty, is it? This method is sure to scare off most casual users, I’d imagine.


 

How to Connect Samsung Pico Projector to a Mac

Samsung Pico Projector
A friend described this device as “an obstinate pony” and I honestly could not agree any more with that description. I enjoy having a larger projection of my screen, but I must admit that it doesn’t come with much instructions, and it is hard to make it behave. For example, if you simply plug it into your mac, it won’t recognise the device. After some hair-pulling and experimenting, I have found the (almost) fool-proof method of connecting it to a Mac. I have no explanation for why this works or what is going on. BUT TRUST ME IT WORKS.

FOLLOW THESE STEPS EXACTLY IN THIS ORDER:

  1. Turn off projector and shut down your mac. Turn it off. Turn off all the power. Disconnect everything!
  2. Start by connecting projector to battery, power, and then use D-Sub Gender cable and connect projector to Mac’s USB drive.
  3. Turn on projector first, before you start up the mac.
  4. Turn on your mac next. Wait for your mac to have completely finished starting up before going to the projector and using its touch buttons to change the selected Input to “PC”.
  5. Projector should be showing the output from your Mac now. Note that you might want to switch to Mirroring if that makes it easier for you to see what is going on…

Creating a Flash app with a PHP script to upload webcam images to WordPress

This month’s challenge is to build a flash application that is able to take a photo, upload it, and insert it into a wordpress post so that posts can be easily edited later. (AND ALL WITHIN ONE OR TWO WEEKS!) I am building this for a museum exhibition that has just opened and it is going to be a kiosk with a large touchscreen that will be moved around from venue to venue, collecting photos of people’s pockets and stuff, and then digitally archiving them, while allowing one to also tag it collaboratively after that.

I am not a PHP developer although I understand PHP sufficiently well in order to make it do what I need to do, so if anyone more technically inclined can tell me if this is an efficient way to proceed with this, I would be glad to hear if there are other ways to do this!

1. Uploading Image to server

I used php’s mktime to generate a unique number for each photo’s filename. Each photo was encoded using JPGEncoder in flash, and then using urlLoader I sent it to my php file, which saved it onto my server with a filename derived from the mktime number.

After that, the posts were pushed to wordpress with the help of Incutio XML-RPC Library for PHP. You can post to wordpress with a php script with the help of xml-rpc. The following examples are my own mashup of bits and pieces found online, the most instructive guide being this entry: WordPress XMLRPC – Posting Content From Outside WordPress Admin Panel

<?php
if(isset($GLOBALS["HTTP_RAW_POST_DATA"])){

$jpg = $GLOBALS[“HTTP_RAW_POST_DATA”];
$img = $_GET[“img”];
$filename = “images/booth_” . mktime(). “.jpg”;
file_put_contents($filename, $jpg);

require_once(“IXR_Library.php.inc”);

$client->debug = true; //Set it to false in Production Environment
$title=’Pocket No. ‘.mktime(); // $title variable will insert your blog title
$body='<img src=”http://yourdomain.org/’.$filename.'”>’; // $body will insert your blog content (article content)
$category=”pocket”; // Comma seperated pre existing categories. Ensure that these categories exists in your blog.
$keywords=”keyword1, keyword2, keyword3″; // Comma Seperated keywords
$customfields=array(‘key’=>’Author-bio’, ‘value’=>’Author Bio Here’); // Insert your custom values like this in Key, Value format

$title = htmlentities($title,ENT_NOQUOTES,$encoding);
$keywords = htmlentities($keywords,ENT_NOQUOTES,$encoding);

$content = array(
‘title’=>$title,
‘description’=>$body,
‘mt_allow_comments’=>0, // 1 to allow comments
‘mt_allow_pings’=>0, // 1 to allow trackbacks
‘post_type’=>’post’,
‘mt_keywords’=>$keywords,
‘categories’=>array($category),
‘custom_fields’ => array($customfields)
);

// Create the client object
$client = new IXR_Client(‘http://yourdomain.org/xmlrpc.php’);
$username = “admin”;
$password = “password”;
$params = array(0,$username,$password,$content,true); // Last parameter ‘true’ means post immediately, to save as draft set it as ‘false’

// Run a query for PHP
if (!$client->query(‘metaWeblog.newPost’, $params)) {
die(‘Something went wrong – ‘.$client->getErrorCode().’ : ‘.$client->getErrorMessage());
}
else
echo “Article Posted Successfully”;

} else{
echo “Encoded JPEG information not received.”;
}
?>

 

2. Retrieving images to put into gallery

This php script will retrieve the last 10 entries in your wordpress and help format them into an xml sheet that you can use in flash to display wordpress entries and images in a flash gallery.

<?php
set_time_limit(0);
require_once("IXR_Library.php.inc"); 
// Echo the header (XML)
header("Content-Type: application/xml;charset=ISO-8859-1");
echo '<?xml version="1.0" encoding="ISO-8859-1"?>' . "rn";
echo "<GALLERY>rn";

// Prepare the XML file
$client->debug = true; // Set it to false in Production Environment

// Create the client object
$client = new IXR_Client(‘http://artsciencepocketbooth.org/xmlrpc.php’);
$username = “admin”;
$password = “password”;
$params = array(0,$username,$password,10); // Last Parameter tells how many posts to fetch

// Run a query To Read Posts From WordPress
if (!$client->query(‘metaWeblog.getRecentPosts’, $params)) {
die(‘Something went wrong – ‘.$client->getErrorCode().’ : ‘.$client->getErrorMessage());
}
$myresponse = $client->getResponse();
$i=0; ?>

<?php
//foreach ($myresponse as $key => $value)
foreach ($myresponse as $res)
{

if($res[‘post_status’]!=”draft”){
//$times = new IXR_Date(); ?>
<image id ='<?php echo $i+1; ?>’ postid='<?php echo $res[‘postid’]; ?>’ title='<?php echo $res[‘title’]; ?>’ keywords='<?php echo $res[‘mt_keywords’]; ?>’/>
<?php $i++; } } ?>
<?php echo “</GALLERY>”; ?>


The resulting XML file will look like this:
Screen shot 2012-03-21 at PM 08

3. Tagging images in gallery

 


 

PHP: Creating automatic XML Sheet from ALL images within a folder

If you need to create an XML sheet from all the photos within one folder, this is a script you can use to create one! It will include any image with a image-related file extension. This is my script which I have customised for my own needs based on notes and tutorials found online.

<?php
// Set which extensions should be approved in the XML file
$extensions = array
(
  'jpg', 'JPG',
  'png', 'PNG',
  'gif', 'GIF',
  'bmp', 'BMP'
);

// Echo the header (XML)
header(“Content-Type: application/xml;charset=ISO-8859-1″);

// Prepare the XML file
echo ‘<?xml version=”1.0″ encoding=”ISO-8859-1”?>’ . “rn”;
echo “<GALLERY>rn”;

// Get the files from the current directory
if (file_exists (“./”))
{
if (is_dir (“./”))
{
$dh = opendir (“./”) or die (” Directory Open failed !”);

// Prepare the images array
$imgs = array();
while ($file = readdir ($dh))
{
// Only get the files ending with the correct extension
if ( in_array( substr($file, -3), $extensions ) )
{
array_push($imgs, $file);
}
}
Closedir ($dh);
}

// Sort the array
sort($imgs);

foreach ($imgs as $img)
{
// Return all images in XML format
echo (‘ <image FULL=”/images/’ . $img . ‘” THUMB=”/images/’ . $img . ‘” />’);
echo “rn”;
}
}
echo “</GALLERY>”;
?>

 


 

OOP & PHP?

there is an interesting article here that argues for OOP in PHP. i admit even in flash sometimes when doing something quick dirty for example i would use global code instead instead of encapsulation. but i guess i find the analogies arguing in favour of OOP rather interesting.

flkinect – simple socket server for kinect and as3 flash

01

flkinect is a Cocoa application by Koji Kimura that allows communication between Flash and Kinect. thought i’d check it out, and realised that it makes it incredibly easy to control some of the simpler functions of the Kinect. i tried it today and it worked instantly. INSTANTLY! WITH ONE CLICK! (Being of the perverse nature, I was almost disappointed it wasn’t harder, or that I didn’t have to understand more in order to make it work!)

Screen shot 2012-03-07 at AM 10

Screen shot 2012-03-07 at AM 10

Not sure why I hadn’t tried this earlier – I supposed I was set on using Windows as I knew that the computer on-site would be Windows, so I never looked back. Another drawback is that i suspect that one will still need to use something like OpenKinect to access the more useful features in kinect. But to get the kinect and flash to basically just communicate in Mac OSX is clearly an absolute breeze, unlike my experience trying to install it for windows…

Traversing XML structures – Loops

I am working on a AS3 project with an XML data sheet and this week I figured out another solution to traversing the structure in a better fashion…

<?xml version="1.0" encoding="utf-8"?>
<TIMELINE>
 <YEAR NAME="2000">
  <ARTICLE>
   <MONTH>January</MONTH>
   <NAME>Lorem Ipsum</NAME>
   <DESCRIPTION>Lorem Ipsum is simply dummy text.</DESCRIPTION>
   <IMAGE>image1.jpg</IMAGE>
  </ARTICLE>
  <ARTICLE>
   <MONTH>December</MONTH>
   <NAME>Ipsum Lorem</NAME>
   <DESCRIPTION>Lorem Ipsum is simply dummy text.</DESCRIPTION>
   <IMAGE>image2.jpg</IMAGE>
  </ARTICLE>  
 </YEAR>
 <YEAR NAME="2003">
  <ARTICLE>
   <MONTH>April</MONTH>
   <NAME>Ipsum Lorem Ipsum</NAME>
   <DESCRIPTION>Lorem Ipsum is simply dummy text.</DESCRIPTION>
   <IMAGE>image3.jpg</IMAGE>
  </ARTICLE>  
 </YEAR>
</TIMELINE>

Basically, in my XML sheet, there were years, and within those years there could be more than one article. Some years had numerous articles. Some years had only one article.

myXML.*.*.length();

When I traced something like this, I would get the total number of articles. I could also traverse all the articles as if the year was not of consequence and all the articles were just being counted and assigned numbers in order from 0 onwards.

myXML.*.ARTICLE[articleNum].DESCRIPTION.toString();

The quandrary was that I needed to be able to scroll through all the articles chronologically but I needed to be able to tell what year it was from, according to the article number (which was now effectively independent from the year), and also be able to find out the article number within that year alone, rather than as a whole.

The solution I came up with: Use a loop to check for number of articles in all previous years, add them up so i can subtract it from the full number of articles. This gives you the article number within that year alone, instead of the article number within the entire “timeline” sheet.

var g:Number = 0;
//myXML.*.ARTICLE[articleNum].parent().childIndex() is year index for that article
while (g < myXML.*.ARTICLE[articleNum].parent().childIndex())
   {
        //myXML.YEAR[g].*.length() is the number of articles in g year
        articleNumContainer += myXML.YEAR[g].*.length();
        g += 1;
   }
// one is added to articleNum cos it actually starts at zero
articleNumberWithinYear = (articleNum + 1) - articleNumContainer;

Flash AS3 and SoThink SWF Decompiler

While saving my work, I had an error and lost the FLA while fortunately, having first published a perfect SWF. If you are saving your work in Flash and Flash crashes before it saves, it often causes your FLA to completely disappear. Always backup your work periodically. Thus I was forced to decompile my own code with SoThink SWF Decompiler to recover my work into an FLA file (for speed) but I also realised that it had decompiled it in a different way from how I had originally coded it. It was easy to figure out and I recovered my entire file within an hour to almost the exact way I had originally coded it. Here are some notes on what I noticed about how it had changed my code in the process of decompiling from swf to fla.

  1. All variables that are randomly declared all over the code are collected and put at the top of the code.
  2. New variables are given names like “_loc_2” or “_loc_3”. Anything with many brackets is also assigned various variable names for different parts of the equation, and then a few of these temporary functions are added up together instead.
  3. All for loops are turned into while loops.
  4. “this.” is added to any objects referred to on the main timeline
  5. It also makes use of the rather uncommon addFrameScript, which adds script to movieclips in AS3. If you leave bits of code outside functions by mistake (eg stage display items), it might group them together in a function called frame1.
  6. It adds the “end function” comment at the end of functions.

Example of typical while loop as decompiled in SoThink:

while (t < max_T_Number)
            {
                // do this
                t = (t + 1);
            }

Example of typical addFrameScript function as decompiled in SoThink:

public function MainTimeline()
        {
            addFrameScript(0, this.frame1);
            return;
        }// end function

function frame1()
{
stage.scaleMode = StageScaleMode.SHOW_ALL;
stage.align = StageAlign.TOP_LEFT;
stage.displayState = StageDisplayState.FULL_SCREEN;
}// end function
}

Although the decompiler is perhaps not the most ethical way of accessing other people’s source code, I think the decompiler still makes an excellent learning tool for flash. When I have more time I will download more swfs or analyse more of my old swfs to see if the conditions listed above are consistently true.

Mushroom

Picture 18
Picture 16

Picture 17

Give me give me moire.

Just some fun sketches I made in flash.