MODULE: Ingestion and schema matching

Many of the most illuminating discussions i have with people on any project is over diagrams.  When someone makes that box connect to that circle via an arrow (or better yet, a dotted line), that is usually when the penny drops for me.

Here are a few of the scribbles that facilitated conversation this week.

The diagram above was used for thinking through the critical path for ingestion new data into DAAO and how it impacts on schema design.

This is what this diagram currently means to me. Our designers and I will be working with CIs (you, imagined reader) to find out how an improved database website will best serve your research needs. Your answers input into modifying the schema design and also guide the design of the website through which you experience the database.  Meanwhile,  Olivia and her editorial/indexing team will mine print new source texts for ingestion – indexing to the new schema.  While Olivia (she’s busy!), the designers and I also go through the new databases, (such as art@base) that we are ingesting and interoperating with (Fuam), and matching and modifying when necessary  non-aligned schema. The next iteration of this diagram should have arrows all folding through these boxes in feedback loops – but the green pen died.

relational database twister

Ross and I were using this twister diagram to talk through how various filters might work to capture new imagined relationships across expanded ‘people’ set. – like artist-curator-designer/design organisation. (There is another re-conceptualisation of ‘organisations’ diagram coming)

This diagram emerged after a very interesting discussion with Mr Snow and Zina from the House of Laudanum – where we discussed metadata registry as way of thinking through ‘truth problem’ with interoperating databases – like Video Art Online (VAO) and DAAO.

There are so many things going in this project: ingesting, connecting, interoperating, reducing ambiguity, enhancing search, standardising, redesigning user experience AND then opening up our data into the miasma of the dataverse so it will be safe in numbers but simultaneously pulling it back again as something defined and redefined.

One of the most personally exciting things about this initial business analysis stage is  gaining a really solid sense of the dynamism and complexity of the landscape in which the DAAO needs to live if it is to survive.  There is so much bending and breaking of things to make things fit and yet there is a drift and flow. At once mechanical and organic, the dataverse is a fascinating place at the moment. New vocabularies emerge every other days to describe the growing, living, dead and dying worlds of data.

This week’s word is agile. (Well it been the word for a while) But as i try and work through critical paths of this project, it is clear that feedback across multiple processes makes linear process difficult. The need to be agile runs from product to process and back.

Welcome to our research blog!

As the project gears up this space will be populated with project updates, discussions between researchers, developers, Design and Art Australia Online (DAAO) staff and other stakeholders.

We have one year to achieve our goal of transforming the Dictionary of Australian Artists Online into Design and Art Australia Online – an expanded, syndicated, database with enhanced content, search, and research functionality -a scalable and sustainable piece of research infrastructure that will help us better understand and document Australian art and design (phew!). So we will be moving quickly.

Shortly, you (our researchers and members of our partner organisations) will start getting a series of short questionnaires devised by our Business Analysis and User Interface Design consultants to probe how the new database would best service your research.

Sometimes we will make short instructional videos on issues such as data mining, syndication, visualisation and how to write an effective user scenario (don’t worry! they are short- usually only 3 sentences long – a type of haiku of use intention) to assist you in some of the questions we might ask.

Also as we transform the database, we want to document its history, so we will also be generating lots of documentation about what we were and where we are going. These videos and diagrams have the added bonus of leaving a little trail of learning and research objects that we are syndicating on our youtube channel http://www.youtube.com/user/DAAOAUS.

Feel free to post, comment, hashtag your twitter feeds #daao or feed your good delicious tags our way. Tag  ‘DAAO’ or ‘Digital Humanities’. (Contact Alice Coltheart at a.coltheart@unsw.edu.au if you want rights to post or if you want us to ingest your feeds)

This is such an exciting project. Olivia, Alice and I are really looking forward to this next year. It’s going to be hectic, but the work will be worth it for what we will build.